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ABSTRACT 



This guidebook sets forth five different methods for 
analyzing curricula. It is designed to be used primarily by teachers, 
curriculum supervisors, and administrators but may also be used by parents, 
students, and community members as they select materials for use in the 
classroom. The analytic methods are: (1) TIMSS Curriculum and Textbook 

Analysis; (2) National Science Foundation (NSF) Instructional Materials and 
Review Process; (3) American Association for the Advancement of Science's 
(AAAS ) Project 2061 Curriculum-Analysis Procedure; (4) California Department 
of Education Instructional Resources Evaluation; and (5) Council of Chief 
State School Officers (CCSSO) State Curriculum Frameworks and Standards Map. 
These five methods were selected for this guidebook because each reflects a 
unique framework and/or set of standards. The selected methods vary in their 
depth of analysis, the time and resources necessary for implementation, 
potential uses, and the types of information and conclusions that can be 
obtained from each. All methods also focus on how instructional materials 
address the needs of diverse learners, including students with disabilities 
and second- language learners. (AIM) 
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Overview of Curriculum Analysis 



Of the many lessons we can learn from the Third International Math- 
ematics and Science Study (TIMSS), one of the most compelling is the 
variation in what is taught to and expected of students. Mathematics and 
science curriculum standards, frameworks, and instructional materials are 
quite different in the countries participating in TIMSS. The conventional 
wisdom holds that mathematics and science are subjects without national 
boundaries. However, TIMSS shows the differences in how mathematics is 
taught and the disparities in expectations for students. 

The TIMSS curriculum study, which is discussed in greater detail later 
in this Guidebook , analyzed the scope and sequence of mathematics and 
science frameworks, standards, and textbooks at every grade level tested in 
the participating countries. The variation is startling, especially at the middle 
to high school years. In addition to the international perspective, an in- 
depth analysis was done for the United States. The report on the United 
States, titled A Splintered Vision: An Investigation of U.S. Science and Mathemat- 
ics Education and summarized at the end of this volume, states, “[S]plintered 
visions produce unfocused curricula and textbooks that fail to define clearly 
what is intended to be taught.” (Schmidt 1996) 

The relationship between curriculum offerings and student achieve- 
ment will be studied as more of the TIMSS data are released. The compari- 
sons that can be made now among the findings of the curriculum study, the 
teacher questionnaires, the teacher video study, and the achievement tests 
lead to the following insights about the curricula offered to our nadon’s 
students: 

■ The content of U.S. eighth-grade mathematics classes is not as 
challenging as that of other countries. 

■ Topic coverage is not as focused in U.S. eighth-grade mathemadcs 
classes as in the classrooms of other countries. (Nadonal Center for 
Education Statistics 1996) 



While the TIMSS studies do not yet draw a conclusive link between 
curriculum and achievement, they reinforce what researchers, policymakers, 
and teachers have long believed about the variable quality and lack of coher- 
ence in U.S. mathemadcs and science curricula. 
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Analyzing Frameworks andTextbooks 

How decisions are made to purchase materials and the criteria used to 
select one set over another have a major impact on what students are ex- 
pected to learn and what teachers teach over many years. While some school 
districts may use a thoughtful approach to match curricular aims and goals 
to instructional materials and textbook series, far too many have neither the 
resources nor the time to make deliberative decisions. The selection of a 
textbook, or any kind of instructional material, needs to be based on a clear 
set of instructional principles and learning goals. Without an in-depth 
analytic review, there is no way of knowing whether the material will actually 
help students learn what is expected. 

In school year 1993-1994, almost $5.5 billion was spent on instructional 
materials by public schools in the United States. The decisions on what to 
purchase were probably made by curriculum committees comprising teach- 
ers, administrators, central office personnel, and community members. In 
some states, selections were made from a list of materials approved by a state- 
level agency or commission. Most states, however, leave these decisions to 
local decision-making bodies who select from the broad marketplace. 

We know from TIMSS and other research the importance of aligning all 
aspects of the education process to ensure that students are provided with 
the best opportunities to achieve to high standards and meet high expecta- 
tions. The interactions among instructional materials, pedagogy, assessment, 
teacher preparation, school capacity, and expectations determine what and 
how much students learn. 

A schoofs curriculum is made up of many different parts. It includes 
textbooks, workbooks, independent assignments, teacher-developed materi- 
als, and state and district frameworks. The most common and recognizable 
piece, however, is the textbook. While there is much debate over how much 
control textbooks have over instruction, we do know that they focus the 
instructional scope and sequence for most teachers and students. 

Most school districts also have curriculum guides or frameworks that 
articulate expectations for what is taught. Recently, many districts also have 
developed standards aligned with state standards or modeled after those 
developed by professional organizations and groups, such as the National 
Council of Teachers of Mathematics, the American Association for the 
Advancement of Science’s Project 2061, and the National Research Council’s 
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National Science Education Standards (NSES). These documents add another 
level of expectations, yet textbooks and other instructional materials are 
rarely aligned with them. 

Given the variety of documents, each articulating its own view of math- 
emadcs and science curricula, a systematic approach to analysis becomes 
even more important. 

Using the Guidebook 

This Guidebook sets forth five different methods of analyzing curricula. 
It is designed to be used primarily by teachers, curriculum supervisors, and 
administrators, and may also be used by parents, students, and community 
members as they select materials to use in classrooms. The analytic methods 
are as follows: 

■ TIMSS Curriculum and Textbook Analysis, 

■ National Science Foundation (NSF) Instrucdonal Materials and 
Review Process, 

■ American Association for the Advancement of Science's (AAAS) 

Project 2061 Curriculum-Analysis Procedure, 

■ California Department of Education Instructional Resources Evalu- 
ation, and 

■ Council of Chief State School Officers (CCSSO) State Curriculum 
Frameworks and Standards Map. 

These five methods were selected for inclusion in the Guidebook because 
each of them is tied to a framework and/or set of standards. This alignment 
was an important criterion for inclusion in the Guidebook . The methods 
selected vary in their depth of analysis, the time and resources necessary, 
their potential uses, and the type of information and conclusions that can be 
gleaned from each. All of the methods pay particular attention to how 
instructional materials address the needs of diverse learners, including 
students with disabilities and second-language learners. 
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Selecting a Method of Analysis 

The section Guide to Using the Methods of Analysis includes more 
information on suggested ways of using each method and brief profiles that 
will help you select the most appropriate method (s) for your purposes. The 
following questions may help you evaluate each of the analytic methods: 

■ Will the results of this analysis be used to select new instructional 
materials or assessments, evaluate the scope and sequence of cur- 
rent materials, and/or determine the alignment between state frame- 

( works and instructional materials? 

■ Will the evaluation be done by teachers or broad-based community 
groups? 

■ Will the results be used by curriculum committees, members of the 
public, and/or administrators? 

It needs to be noted that this Guidebook does not examine the role of 
teachers and the efficacy of different instructional practices. While most of 
the methods of analysis do include pedagogy and ease of use as aspects to be 
examined, the focus is not on teaching per se. The capacity of the school 
system, including teachers’ instructional approaches and professional prepa- 
ration, influences how instructional materials come alive in the classroom. 

Regardless of which analytic methods are used, every school or district 
needs to ask itself what students are expected to know, when they are ex- 
pected to know it, and what materials are used. The TIMSS results have 
shown that U.S. curricula are generally not as rigorous as those in Germany 
and Japan. All eighth graders in these countries spend most of the year 
studying algebra and geometry, while most U.S. students study these subjects 
later. Only by looking deeply and systematically at what is expected and 
taught will we be able to raise student achievement. 

Reviewing curricula as a whole needs to be part of every school’s deci- 
sion-making portfolio. These procedures inform important decisions that 
are made regularly in schools throughout the country. 
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Guide to Using the Methods of Analysis 

Purpose 

The purpose of this Guidebook is to present different ways of analyzing 
instructional materials for mathematics and science, including curricula, 
curriculum frameworks, textbooks, instructional modules, classroom activi- 
ties, or teachers’ guides. Each of these methods of analysis provides different 
sets of information about instructional materials, ranging from descriptions 
and analyses of content and structure, to evaluations of their potential for 
leading to specific learning goals. Thus, whether you are designing, review- 
ing, or selecting materials, some components of this module are relevant. 

Uses of the Analysis Methods 

As you develop, review, or select your instructional materials, there are 
several ways you might use these different methods: 

■ As a general reference — Each of these methods emphasizes certain 
qualities, such as balance of topics, student skills and behaviors 
encouraged, or connection to specific curriculum frameworks. 

These qualities should be kept in mind as you conduct your own 
curriculum development and review processes. 

■ Adapted to local needs — These methods have two components, a 
methodology (process of analyzing curriculum materials) and a 
frame of reference (frameworks or standards to which they are 
compared). Even if you have a different frame of reference (such as 
your own state’s curriculum framework), you will be able to adapt 
the methodology to your own needs. 

■ As presented — If the frame of reference is relevant to your needs, 
you may wish to use a particular method as presented. In some 
cases, the material included in this Guidebook is sufficient for you to 
conduct the analysis yourself, while in other cases you may need 
additional resources or assistance from an outside organization. 

The methods of analysis in this Guidebook examine instructional materi- 
als before they are put into use by schools and teachers. They examine the 
content and quality of the materials as a set of documents that articulate a 
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course of study and an instructional approach. In fact, the primary goal of 
one of the methods, the AAAS Project 2061 Curriculum-Analysis Procedure, 
is to judge the materials in terms of the likelihood that they would contribute 
to the attainment of specific learning goals. If questions of implementation 
and impact are also among your concerns, you should consider additional, 
more direct methods of assessing these factors. 

As you decide which methods to use, you must be sure that they are 
appropriate to your needs. Individual methods examine particular aspects of 
instructional materials. For example, the TIMSS Curriculum and Textbook 
Analysis looks at the topics to be presented but does not examine the accu- 
racy of the material. The CCSSO Curriculum Frameworks and Standards 
Map looks at broad frameworks and standards but not at instructional 
materials. 

Each of the analydc models should be reviewed carefully before any 
decisions are made to use one instead of another. Each reflects a different 
perspective on curriculum materials, requires different amounts of time to 
complete, and may require training or additional assistance. 

Guide to Contents 

This section provides a matrix of the five analytic methods presented in 
this Guidebook and a brief description of each. The brief descriptions answer 
key questions about each method and should be used as an overview and to 
compare the methods. 

The section TIMSS Curriculum and Textbook Analysis includes a 
background and overview of the process and how it was used. 

The Other Curriculum-Analysis Methods section presents four addi- 
tional methods of analysis for examining instructional materials: 

NSF Review of Instructional Materials — Developed by the National 
Science Foundation to review its funded comprehensive middle- 
school science and mathematics projects. 

AAAS Project 2061 Curriculum-Analysis Procedure — Developed 
by Project 2061 at the American Association for the Advancement of 
Science for reviewing a variety of prepared curriculum materials. 
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California Instructional Resources Evaluation — Developed by the 
California Department of Education for its Instructional Resources 
Adoption Process. 

CCSSO Curriculum Frameworks and Standards Analysis — Devel- 
oped by the Council of Chief State School Officers to describe state 
mathematics and science standards and curriculum frameworks. 

For each method of analysis, the Guidebook contains background infor- 
mation, a description of the analysis process, and, in some cases, forms or 
other instruments for conducting the analysis. There is also an overview for 
each method of analysis that summarizes it in terms of several key questions. 
The methods of analysis are outlined in the matrix that follows on page 9. 

Key Matrix and Overview Headings Include: 

■ What issues does it address? Not all of the methods look at the 
same issues or answer the same questions. Some of the methods 
review only content and seek to answer fairly focused questions, 
such as “Which curriculum topics are addressed, and how much 
attention is devoted to each?” Others are more in depth, looking at 
issues of pedagogical approach and answering such questions as “Is 
this material acceptable for use?” 

■ What materials does it examine? Some methods are designed to 
look only at topics and subtopics to be addressed. Others look at 
textbooks and teachers’ gLiides, which, in addition to an outline of 
topics, provide student activities and instructional strategies. Some 
are designed to look at any of these individually or as sets. 

■ What is the frame of reference? All methods involve comparisons 
to a particular framework or set of standards for mathematics and 
science, such as a state framework or the National Council of Teach- 
ers of Mathematics’ (NCTM) Curriculum and Evaluation Standards for 
School Mathematics , the National Science Education Standards (NSES), 
or the Benchmarks for Science Literacy from the AAAS Project 2061. 
Although you may base your curriculum on another framework, the 
method of analysis will still prove useful. 
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What is the analysis process? Most methods involve the use of 
review forms or protocols. The processes differ, however, in the 
level of detail at which materials must be examined and whether 
they require teams of reviewers. 

What resources does this type of analysis require? The process of 
materials analysis may be a major undertaking, requiring large 
amounts of staff time and special funds. In some cases, the people 
performing the analysis should possess particular backgrounds, 
such as in teaching science, or familiarity with a particular frame- 
work. Also, using a specific method may require the purchase of 
additional resources, such as training materials, guidebooks, or 
evaluation forms. 

Does this analysis require outside assistance or special training? 

To use some of the methods, it may be necessary to send materials 
to an outside organization for analysis or to consult with the devel- 
opers. In some cases, special training for using the method is 
available. 

What are the potential uses at the local level? Each of the meth- 
ods will provide different types of information and assist the user in 
drawing different types of conclusions. 

Contact information. The people listed are available to answer more 
detailed questions about the methods and about using them in your 
school or district. 
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Overview of TIMSS Curriculum and Textbook Analysis 
Background 

The TIMSS project provides a comprehensive look at mathematics and 
science educadon in different countries. It focuses on three areas: intended 
curricula (what is supposed to be taught), implemented curricula (what is 
actually taught and how), and achieved curricula (what is learned). The 
curriculum and textbook analysis process presented in this module was 
designed to address the first and second areas, intended curricula and 
implemented curricula. TIMSS researchers used this process to compare 
curriculum documents across almost 50 countries. Prior to TIMSS, interna- 
tional comparisons of curricula relied primarily on the opinions of experts. 
The TIMSS curriculum and textbook analysis provides a systematic method 
of analyzing and comparing original curriculum documents using an inter- 
national framework. It should be emphasized that it is an analytic tool, 
meaning that it does not make judgments regarding what is good or bad, but 
rather seeks to illustrate the similarities and differences between mathematics 
and science curricula in different countries. 

What issues does it address? 

Breadth and depth of content (topics, e.g., relationships of common 
decimal fracdons) and performance expectations (thinking skills, e.g., for- 
muladng and clarifying problems and situadons). Within a grade, it can help 
you determine whether you are covering many topics, but with brief atten- 
don to each, or if your curriculum is more focused, and whether the perfor- 
mance expectations are balanced. You can also see how the sets of topics and 
performance expectations change from grade to grade. You can compare 
your curriculum “profile” to those of schools in other countries. 

What materials does it examine? 

Curriculum guides and textbooks. 

What is the frame of reference? 

The TIMSS Curriculum Framework and curriculum profiles of other 
countries. 
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What is the analysis process? 

Items in the materials to be examined are placed into frameworks of 
categories and subcategories of content topics and performance expecta- 
tions. The results are then analyzed to show which topics and performance 
expectations are addressed, under which broad areas they fall, the amount of 
time devoted to each, and the total number of topics and performance 
expectations. 

What are the potential uses at the local level? 

Although the analysis methods presented here were developed to com- 
pare curricula across countries, they have several potential uses for local 
schools and districts, for example: 

■ Benchmarking to a country involved in the TIMSS study, a group 
of countries, or an international composite. This is useful to 
schools, districts, and communities interested in knowing how their 
curricula compare with those of a country, or set of countries, with 
high levels of student mathematics and science achievement, or 
with countries they see as having high standards. 

■ Comparisons of topic coverage profiles with local goals and 
priorities. While the TIMSS analysis did not make judgments 
about whether a curriculum was good or bad, these may be the 
types of judgments schools and districts would like to make. With a 
TIMSS-style profile of topic coverage and performance expecta- 
tions, schools and districts can determine where there are undesir- 
able gaps/ overlaps in their science and mathematics curricula across 
grade levels. The results would help them see if they have too many 
topics at a given grade level and if they are not giving enough 
attention to a particular topic. 

■ Substitution of another framework for the TIMSS framework. 

Schools and districts may use similar methods of Topic Trace 
Mapping and Document Analysis, but with a different curriculum 
framework, such as a state framework or the NSES or NCTM stan- 
dards. Only when the TIMSS framework is used, however, can 
districts compare themselves to other countries or international 
benchmarks. 
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Does this analysis require outside assistance or special training? 

Performing this analysis as was done for the TIMSS project requires the 
assistance of the TIMSS staff at Michigan State University. Contact informa- 
tion is provided below. 

What resources does this type of analysis require? 

The TIMSS staff charges a fee for conducting the analysis. The fee 
depends on the extent of analysis, including such variables as district or 
school size, number of textbooks, and curriculum documents to be analyzed. 



Contact information: William H. Schmidt, Director 

U.S. National Research Center 
TIMSS Curriculum Analysis Project 
Michigan State University 
College of Education 
457 Erickson Hall 
East Lansing, MI 48824 
Telephone: (517) 353-7755 
E-mail: bschmidt@pilot.msu.edu 
World Wide Web: http://ustimss.msu.edu 
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Overview of National Science Foundation 
Review of Instructional Materials for Middle School Science and 
Mathematics and Framework for Review: Instructional Materials for Middle 
School Mathematics 

Background 

In 1996, the National Science Foundation (NSF) undertook a study of 
comprehensive (at least one year) instructional materials for science in the NSF 
portfolio encompassing the middle-school years. This middle-school review 
was the first effort to examine a range of projects for a particular set of grades. 

The major goals of NSF are to (1) provide the field with high-quality 
instructional materials that incorporate the best research on teaching and 
learning; (2) include accurate science and the active participation of scientists 
in the development process; and (3) have undergone an extensive pilot and 
field-test process. Support of such materials enables teachers, schools, and 
districts to have access to materials that provide students with experiences 
that lead to an understanding and mastery of scientific concepts and pro- 
cesses. 

What issues does it address? 

Alignment of materials with NSES and NCTM standards. Criteria 
include accuracy of the material presented, coverage of topics, pedagogi- 
cal design (e.g., Do the materials provide for conceptual growth? How do 
they engage students?), assessment methods, support for implementation 
(e.g., Do the materials provide information on available resources and 
necessary support structures?), and equity (e.g., Are the materials likely to 
be interesting, engaging, and effective for underrepresented and 
underserved students?). 

What materials does it examine? 

Comprehensive sets of instructional materials — for example, a coordi- 
nated package of student books, hands-on materials, multimedia materials, 
assessments, and a teacher’s guide designed to cover one or more years of 
instruction. 
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What is the frame of reference? 

The NSES and NCTM standards. 

What is the analysis process? 

Teams of practicing scientists or mathematicians, educators, and assess- 
ment and implementation specialists review materials using a common review 
framework (included in this module). Team members assigned with specific 
portions of the materials review them individually, assign scores using the 
forms, and then meet to discuss their individual evaluadons and develop a 
consensus assessment. 

What are the potential uses at the local level? 

The analytic method developed by NSF reflects three recognized dimen- 
sions of good instructional materials: 

■ Alignment with National Science Education Standards or National 
Council of Teachers of Mathematics Standards. This is useful to 
schools as they examine the coherence and alignment of their 
curricula with recognized national standards. Where local standards 
and frameworks are unique, schools and districts may wish to use 
the NSF review process but link it to their own standards. It pro- 
vides valuable information on the rigor of their curricula and in- 
structional materials. 

■ Sound pedagogical design and logical development of conceptual 
understanding. The NSF model asks evaluative questions that 
relate content and design and focuses on whether instructional 
materials reflect conceptual growth and provide students with 
opportunities to gain a better understanding of the information. 

■ Provision for ongoing student assessment. Materials that embed 
assessments within the instructional approach provide both stu- 
dents and teachers with a better picture of how well students are 
learning. 

Does this analysis require outside assistance or special training? 

Training is highly recommended to familiarize participants with the 
process and review criteria. 
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What resources does this type of analysis require? 

For each team member involved, approximately five working days, 
broken down as follows: 



1 day Team training 

2 days Individual review of materials 

(can be spread over a longer period of time) 

2 days Team discussion of individual reviews and 
consensus assessment 



Contact information : Janice Earle, Ph.D. 

Program Director 
Elementary, Secondary, and 
Informal Education 
National Science Foundation 
4201 Wilson Boulevard, Room 885 
Arlington, VA 22230 
Telephone: (703) 306-1614 
Fax: (703) 306-0412 
E-mail: jearle@nsf.gov 
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Overview of American Association for the Advancement of Science (AAAS) 
Project 2061 Curriculum-Analysis Procedure 

Background 

With the growing consensus on what all students should know and be 
able to do in science, mathematics, and technology, educators now need a 
reliable method for identifying curriculum materials that will help students 
achieve those learning goals. Working with hundreds of K-12 teachers, 
materials developers, cognitive researchers, and scientists, Project 2061 has 
developed a systematic procedure for analyzing curriculum materials for 
alignment to specific learning goals. This procedure was created under a 
grant from the National Science Foundation and has been tested under field 
conditions by teachers at six sites around the country. 

What issues does it address? 

The Project 2061 Curriculum-Analysis Procedure focuses on three 
central concerns: What content does the curriculum material target? How 
well does that content align with specific learning goals such as benchmarks 
or standards? Does the material provide appropriate instructional strategies 
to help students learn the intended content? Project 2061 ’s procedure offers 
a systematic approach to answering these questions. 

The Project 2061 procedure identifies which specific learning goals the 
content of the material addresses, and estimates the effectiveness of its 
explicit instructional approaches for those specific learning goals. This is in 
contrast to procedures that make separate judgments about general topic 
coverage and general instructional quality. 

What materials does it examine? 

K-12 curriculum materials — ranging from short units to multiyear 
programs, including textbooks and their accompanying teachers’ guides — 
that deal with the natural and social sciences, mathematics, and technology. 
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What is the frame of reference? 

Project 2061’s Benchmarks for Science Literacy, the National Research 
Council’s National Science Education Standards , the National Council of Teach- 
ers of Mathematics’ Curriculum and Evaluation Standards for School Mathemat- 
ics , and, when they become available, ITEA Standards ( Technology for All 
Americans). States can use their own frameworks or standards as the frame of 
reference, provided the learning goals are specific. 

What is the analysis process? 

Working as a team, reviewers first identify a sample of plausible specific 
learning goals against which to compare curriculum material. Next, they 
apply sets of analytical criteria to judge how well the material is aligned to the 
learning goals. This includes judgments about (1) how well the material’s 
content matches the specific learning goals and (2) the extent to which the 
material’s instructional strategy promotes student learning of the content. 

In a final report, reviewers summarize their findings, develop profiles for the 
materials, and present their conclusions. 

What are the potential uses at the local level? 

The AAAS Project 2061 Curriculum-Analysis Procedure provides dis- 
tricts/ schools with a standards-based approach to evaluating instructional 
materials. For example, a school or district can use the procedure to consider 
the following: 

■ Appropriateness of the content. When selecting new materials or 
examining current offerings, a district will want to evaluate the 
extent to which textbooks and other curriculum materials are con- 
sistent with national, state, or local standards/frameworks and in line 
with learning goals for students at specific grade levels. 

■ Utility of the instructional design. Paying attention to the under- 
lying instructional strategy of curriculum materials will help the 
district or school decide whether the materials will help students 
meet its expectations. 

Does this analysis require outside assistance or special training? 

Depending on audience and purpose, at least three to five days of train- 
ing are recommended, which includes practice analyzing actual materials. 

Some groups might want additional time to familiarize themselves with Bench- 
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marks for Science Literacy or other standards documents to which they want 
materials to align. Project 2061 will publish a training manual in 1998 (“Re- 
sources for Science Literacy: Curriculum Evaluation”) as part of a CD-ROM/ 
print tool, which will provide step-by-step instructions for the procedure, along 
with case studies of analyzed materials. In the meantime, interested educators 
can contact Project 2061 for information about training opportunities. 

What resources does this type of analysis require? 

Personnel — At least two individuals should review the material indepen- 
dendy so that they can compare and reconcile results. The number of reviewers 
required naturally depends on the magnitude and scope of what is being 
reviewed. For looking at units, two individuals might suffice, but for looking at 
larger curriculum components, across grades or disciplines, teams that collec- 
dvely have the necessary subject-matter and grade-level expertise are needed. 

Materials — In addition to the curriculum material under consideration, 
reviewers will need Project 2061 s Resources for Science Literacy: Curriculum 
Evaluation (available in 1998). Resources will also provide detailed compari- 
sons of Benchmarks for Science Literacy and national standards in science, 
mathematics, and technology. Reviewers will also need copies of Benchmarks 
for Science Literacy , Technology for All Americans , and the relevant set of na- 
tional (or local) standards against which they want to evaluate materials. 

Many reviewers will also find Benchmarks for Science Literacy on Disc very 
useful, with its search feature and sample growth-of-understanding maps. 

Contact information: Mary Koppal 

Communications Director 
Project 2061 

American Association for the 

Advancement of Science 

1333 H Street, NW 

PO. Box 34446 

Washington, DC 20005 

Telephone: (202) 326-6666 

Fax: (202) 842-5196 

E-mail: project2061@aaas.org 

Worldwide Web: http://www.aaas.org/project 2061 
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Overview of California Department of Education 
Curriculum Frameworks and Instructional Resources: 

Mathematics Instructional Materials 
Evaluation Instrument and Rating Form 

Background 

For use in grades one through eight, the California State Board of 
Educadon has the consdtudonal authority to adopt textbooks, based on their 
consistency with the State Board’s curriculum frameworks. The state’s 
Curriculum Commission recommends and the State Board appoints a panel 
of individuals to conduct an in-depth review of the textbooks. After review- 
ing the textbooks, the panel makes recommendadons to the Curriculum 
Commission. This Guidebook contains the evaluadon form for use by panels 
when evaluadng mathemadcs textbooks in grades K-8. 

What issues does it address? 

Alignment of instrucdonal resources with California state standards for 
mathematics. Criteria for mathemadcs include: mathemadcal content, 
program organization and structure, the work students do, student diversity 
(how the program deals with diversity in backgrounds, abilities, and inter- 
ests), integration of assessment and instrucdon, and support for the teacher. 

What materials does it examine? 

Designed for textbooks, it can also be used for other instructional 
resources, such as technology-based resources or manipuladve kits. 

What is the frame of reference? 

Mathematics Frameworks for California Public Schools , Kindergarten Through 
Grade Twelve. 

What is the analysis process? 

Panels of reviewers evaluate materials using a common form (included 
in this Guidebook ). Team members review materials, assign scores individually 
using the forms, and then meet to discuss their individual evaluadons and 
develop a consensus assessment. 
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What are the potential uses at the local level? 

The methods developed for the state of California provide a technique 
for rating different sets of materials with a numeric score. By using this 
technique, a school or district can: 

■ Compare different sets of instructional materials to a scoring 
rubric. When examining different textbook series, a school or 
district will be able to compare and contrast the materials using a 
standard weighted rubric. 

■ Establish different weights for review criteria. This method of 
analysis provides weighted criteria across six areas. The relative 
percentages can be changed to emphasize different perspectives. 

■ Substitute another framework. The areas of emphasis and relative 
weights were determined by the California mathematics framework. 

If another framework is used, it should be closely examined and the 
relative weights aligned to the emphasis of the other framework. 

Does this analysis require outside assistance or special training? 

Experts on the California frameworks (or on those frameworks used in 
place of them) should meet with panel members to familiarize them with the 
frameworks. 

What resources does this type of analysis require? 

For each team member involved, approximately five working days, 
broken down as follows: 

1 day Team training 

3 days Individual review of materials 

(can be spread over a longer period of time) 

1 day Team discussion of individual reviews and 
consensus assessment 
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Contact information'. Curriculum Frameworks and 
Instructional Resources Office 
California Department of Education 
721 Capitol Mall, Sixth Floor 
P.O. Box 94472 
Sacramento, CA 94244-2720 
Telephone: (916) 657-3023 
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Overview of Council of Chief State School Officers (CCSSO) 

State Curriculum Frameworks and Standards Map: 

Definitions of Categories and Concepts 

Background 

The CoLincil of Chief State School Officers (CCSSO) conducted a com- 
prehensive study of the status, characteristics, and quality of state curriculum 
frameworks and standards in mathematics and science. An initial report in 
1995 described the process of development of state frameworks and standards 
in the 1990s. In 1997, CCSSO released a second report addressing new 
frameworks and standards completed by the end of 1996, which included a 
total of 32 states. A key component of the study was a “Conceptual Map of 
State Frameworks and Standards.” The map categorized and described state 
frameworks and standards documents across 14 concepts or “elements.” 

What issues does it address? 

The findings are intended for use in identifying states (or districts) with 
specific standards in mathematics and science, in finding information in the 
documents on how standards can be applied and used with schools, and in 
providing examples of the different ways states have addressed development 
of frameworks and standards. 

The mapping elements include sources of information; development 
process; funding; pages; year; number and types of content standards; 
number and types of benchmarks/indicators; related state documents; 
communication methods; equity and inclusion; pedagogy; assessment; 
professional development; and the use of technology, materials, and texts. 

What materials does it examine? 

Curriculum frameworks, content standards, other state and district 
guidance on curriculum development. 

What is the frame of reference? 

A set of elements developed by the CCSSO expert panel, based on 
national professional standards publications, international frameworks, and a 
reading of a sample of state frameworks and standards. 
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What is the analysis process? 

Definitions and categories are specified for each element. Documents 
are thoroughly read by two trained research staff who code the documents 
against the analysis categories. Codes, examples, brief descriptions, and state 
definitions are entered into a database. 

What are the potential uses at the local level? 

This method of analysis was created to look across frameworks and 
develop a comprehensive model of state frameworks and standards. At the 
school or district level, potential uses include: 

■ Comparison of local standards/frameworks with state standards. 

This is useful to local communities developing their own frame- 
works or standards for seeing how they compare within their own 
states or to other neighboring states. 

■ Analysis of state standards and frameworks. A school, district, or 
consortium of districts may need to analyze the state standards 
prior to evaluating a set of instructional materials. This will be 
particularly important if a state assessment is in progress. 

Does the analysis require outside assistance or special training? 

Training for two to three days is needed. Research staff should have 
expertise in mathematics and/or science education. For assistance, see 
contact information. 

What resources does this type of analysis require? 

The analysis of each state document requires, on average, about three 
days. This assumes the staff has been trained, has been oriented to the task, 
and has done a pilot analysis. Two staff analyze the same document, and 
another staff person monitors any differences in categorization and coding. 
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Contact information: 
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Rolf K. Blank, Ph.D. 

Director of Education Indicators 
Council of Chief State School Officers 
One Massachusetts Avenue, NW, Suite 700 
Washington, DC 20001-1431 
Telephone: (202) 408-5505 
Fax: (202) 789-1792 
E-mail: rolfb@ccsso.org 
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TIMSS Curriculum and Textbook Analysis 

Introduction 

u International achievement tests tell us what and how well students in the 
United States have learned compared to students in other countries , but are 
all U.S. students expected to learn the sam,e things ? Are our students 
expected to learn more or less material than other students , and when are 
they expected to learn it?” 

These are the types of questions the TIMSS curriculum analysis at- 
tempted to answer. TIMSS is the third large-scale international study of 
mathematics and science education conducted by the Internadonal Associa- 
tion for the Evaluation of Educational Achievement (IEA). The first two 
studies focus almost exclusively on comparing student achievement. While 
the achievement data indicate how students compare to each other on a 
common examination, they leave unanswered whether all students study the 
same material or are taught in similar settings with similar techniques. 
Therefore, TIMSS took a broader look at the factors contribudng to achieve- 
ment, particularly curriculum, instrucdon, and the environments of students 
and teachers. One purpose was to shed more light on achievement data, but 
the study also was designed to provide a much more sophisticated under- 
standing of educadonal pracdces. The goal was not to judge which coun- 
tries had “good” or “bad” curricula and instrucdonal pracdces, but rather to 
analyze and compare them. 

The TIMSS Curriculum Frameworks 

The first step in comparing mathematics and science educadon across 
countries was to develop a common, internadonal frame of reference for 
talking about learning goals. The results are the TIMSS curriculum frame- 
works for mathemadcs and science (see Figure 1 on page 29). These frame- 
works cover all of the years of schooling. They consider three aspects of 
curricula: (1) content — subject matter topics, (2) performance expecta- 
tions — what students are expected to do with pardcular content, and (3) 
perspecdves — overarching themes connecdng subject matter to its place 
among the disciplines and the everyday world. By using this classification 
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system, any curriculum component can be described by a “signature” con- 
sisting of categories and subcategories from each of the three aspects. Again, 
the frameworks are not meant to serve as statements regarding what a cur- 
riculum should include, but to help describe what a curriculum does include. 

Content 

The most immediate questions regarding curriculum center around 
whether students from different countries are studying similar material and, 
if not, how the curricula differ. Figure 1 lists the content categories for both 
mathematics and science. In science, the eight content categories are further 
divided into 47 subcategories with 66 subordinate subcategories. In math- 
ematics, the 10 categories are further divided into 29 subcategories and 20 
subordinate subcategories. The complete content frameworks for mathemat- 
ics and science can be found in Figures 6 and 7 on pages 35 and 38, respec- 
tively. Comparisons using the content frameworks help identify whether 
students in different countries are expected to study similar material. 

Performance Expectations 

In addition to the topics covered, TIMSS researchers investigated 
expectations of what students are to do with the knowledge they acquire. As 
the framework indicates, performance expectations range from understand- 
ing information to applying it in theorizing, problem solving, and investiga- 
tion. Comparing different curricula to the performance expectations frame- 
work can help identify which curricula place greater emphasis on under- 
standing and which tend to emphasize application. 

Perspectives 

The goal of the perspectives framework is to identify broader goals for 
teaching mathematics and science than either acquisition of knowledge or 
development of skills. These goals include developing positive attitudes 
toward the subject matter and careers in the field. The materials in this 
module focus primarily upon content and performance expectations. 
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Figure 1. The TIMSS Science and Mathematics Curriculum Frameworks 
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Analyses Performed and Examples of Questions They Answer 

Using this framework, TIMSS researchers conducted two types of 
analyses to compare mathematics and science curricula. These analyses were 
intended to answer questions about topic inclusion, curriculum depth and 
breadth, and relative emphasis placed on various topics. TIMSS researchers 
also conducted a survey of seventh- and eighth-grade teachers in Japan, 
Germany, and the United States. 

Topic Trace Mapping (of content) 

In this procedure, a panel of curriculum experts in each country identi- 
fied the grade levels at which particular topics from the TIMSS framework are 
included in their country’s curriculum frameworks. (It should be noted that, 
unlike the United States, most countries have national curriculum guidelines.) 
Doing so allowed researchers to draw a “map” of all grade levels showing when 
each topic enters and leaves the curriculum and how long it stays. It also 
allowed comparisons of the total number of topics included in the curriculum 
at each grade level. The data can be used to answer a large number of research 
questions regarding both the “life” of topics in the curriculum over all years of 
schooling and topic “profiles” of each grade level. Comparisons can be made 
with specific countries or as an international composite. Following are three 
questions addressed in the study and the results. 

(1) How many topics do we plan to cover at each grade level? 



Figure 2. Number of Mathematics Topics Intended 
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(2) What is the number of topics added and dropped at each grade level? 



Figure 3. Number of Science Topics Added and Dropped at Each Grade Level 
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(3) How long do we plan to continue study of a topic? (In how many grades 
is it addressed?) 

Figure 4. Average Mathematics Topic Duration 
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Figure 5. Curriculum Coverage for 
Selected Science Topics Across Student Ages 

• topic covered in curriculum ■ topic emphasized in curriculum 
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Example 2: Organs & Tissues 
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Example 3: Reproduction of Organisms 
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Example 4: Chemical Properties of Matter 
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10 11 


12 


13 


14 


15 


16 


17 18 


Country 


















France 








• 


• 


■ 


■ 


■ 


Japan 




■ 


• 


■ 


■ 


• 


■ 


■ 


Norway 


















Spain 




• 


• 


■ 


• 


■ 


• 


■ 


Switzerland 
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■ 


■ 




United States 
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• 
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ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



Document Analysis 

The TIMSS study focuses on three different student populations: 
Population 1, students in the two consecutive grades with the majority of 9- 
year-olds; Population 2, students in the two consecutive grades with the 
majority of 13-year-olds; and Population 3, students in the final year of 
secondary school. For Populations 1 and 2, and for students in Population 3 
specializing in mathematics and physics, researchers conducted an in-depth 
analysis of curriculum guides and textbooks using a two-step process: 

(1) Researchers divided each document into units — major structural 
components — and blocks — smaller segments within units. 

(2) Each of these units and blocks was then assigned content, perfor- 
mance expectation, and perspective category codes from the frame- 
works. 

Several measures were taken to ensure uniformity and reliability across 
the large number of teams involved in the coding process. These included 
the development of detailed manuals on the procedures, intensive training 
sessions, and an initial quality-assurance phase in which teams were not 
allowed to begin their coding until they had been evaluated satisfactorily in a 
trial coding exercise by an international panel of referees. 

With the documents fully coded, researchers were then able to describe 
and compare the science and mathematics curricula of different countries in 
terms of the topics and performance expectations included, their relative 
emphases, and the total number of content topics. Following are two ques- 
tions addressed in the study and the results. 

Which topics do toe plan to cover ? 

Figures 6 and 7 show the topics from the TIMSS frameworks included 
in the curriculum guides and textbooks of the United States and a composite 
of the other participating countries. A V” means that the topic was listed in 
at least 70 percent of the curriculum guides or textbooks. 
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Figure 6. Topics Covered in Mathematics Curriculum Guides and Textbooks 



TIMSS Framework 


Population 1 


Population 2 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 










Non-Algebra 


Algebra 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


1.1 Numbers 

1.1.1 Whole numbers 


1.1. 1.1 Meaning 


/ 


/ 


/ 


/ 








/ 




/ 


1.1. 1.2 Operations 


/ 


/ 


/ 


/ 






/ 


/ 


/ 


/ 


1.1. 1.3 Properties of operations 


/ 


/ 




/ 




/ 




/ 




/ 


1 .1 .2 Fractions and decimals 


1.1. 2.1 Common fractions 


/ 


/ 


/ 


/ 




/ 


/ 


/ 


/ 


/ 


1.1. 2.2 Decimal fractions 


/ 




/ 


/ 




/ 




/ 




/ 


1.1. 2.3 Relationships of common 
and decimal fractions 








/ 




/ 


/ 


/ 


/ 


/ 


1.1. 2.4 Percentages 












/ 


/ 


/ 


/ 


/ 


1.1. 2.5 Properties of common 
and decimal fractions 






















1.1.3 Integer, rational, 
and real numbers 


1.1. 3.1 Negative numbers, inte- 
gers, and their properties 








/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.1. 3. 2 Rational numbers 
and their properties 










/ 


/ 




/ 




/ 


1.1. 3. 3 Real numbers, their sub- 
sets, and their properties 










/ 






/ 




/ 


1.1.4 Other numbers 

and number concepts 


1.1. 4.1 Binary arithmetic and/or 
other number bases 
















/ 






1. 1.4.2 Exponents, roots, 
and radicals 










/ 


/ 




/ 




/ 


1.1. 4.3 Complex numbers 
and their properties 


















/ 

/ 


/ 


1.1. 4.4 Number theory 








/ 








/ 




/ 


1.1. 4.5 Counting 








/ 








/ 







ERK 

fariKnim i lrt i i GUIDEBOOK TO EXAMINE SCHOOL CURRICULA 




35 














ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



Figure 6. Topics Covered in Mathematics Curriculum Guides and Textbooks (continued) 



TIMSS Framework 


Population 1 


Population 2 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 
















Non-Algebra 


Algebra 1 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


1 .1 .5 Estimation and number sense 1 


1.1. 5.1 Estimating quantity 
and size 








/ 






/ 


/ 


/ 




1.1. 5. 2 Rounding and 

significant figures 








/ 






/ 


/ 


/ 




1.1. 5.3 Estimating computations 






/ 


/ 






/ 


/ 


/ 




1.1. 5.4 Exponents and 

orders of magnitude 














/ 




/ 




1 .2 Measurement 


1.2.1 Units 


/ 


/ 


/ 


/ 




/ 


/ 


/ 


/ 




1.2.2 Perimeter, area, and volume 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.2.3 Estimation and error 






/ 


/ 






/ 


/ 


/ 




1.3 Geometry: position, 
visualization, and shape 


1.3.1 Two-dimensional geometry: 
coordinate geometry 


/ 




/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.3.2 Two-dimensional geometry: 
basics 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.3.3 Two-dimensional geometry: 
polygons and circles 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1 .3.4 Three-dimensional geometry 






/ 


/ 


/ 


/ 


/ 


/ 


/ 




1.3.5 Vectors 














/ 




/ 




1 .4 Geometry: symmetry, 
congruence, and 
similarity 


1.4.1 Transformation 


/ 




/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.4.2 Congruence and similarity 






/ 


/ 


/ 




/ 


/ 


/ 


/ 


1.4.3 Constructions using 
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Figure 6. Topics Covered in Mathematics Curriculum Guides and Textbooks (continued) 



TIMSS Framework 


Population 1 


Population 2 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 










Non-Algebra 


Algebra 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


1 .5 Proportionality 


1.5.1 Proportionality concepts 








/ 


/ 




/ 


/ 


/ 


/ 


1.5.2 Proportionality problems 










/ 


/ 




/ 




/ 


1.5.3 Slope and trigonometry 
















/ 




/ 


1.5.4 Linear interpolation 
and extrapolation 






















1.6 Functions, relations, 
and equations 


1.6.1 Patterns, relations, 
and functions 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.6.2 Equations and formulas 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.7 Data representation, 
probability, and statistics 


1.7.1 Data representation 
and analysis 




/ 


/ 


/ 






/ 


/ 


/ 




1 .7.2 Uncertainty and probability 






/ 


/ 






/ 


/ 


/ 




1.8 Elementary analysis 


1.8.1 Infinite processes 






















1.8.2 Change 






















1.9 Validation and structure 


1.9.1 Validation and justification 








/ 








/ 




/ 


1.9.2 Structuring and abstracting 








/ 








/ 




/ 


1.10 Other content 


/ 


/ 


/ 


1.10.1 Informatics 






















Total number of topics 


11 


9 


18 


29 


16 


18 


24 


37 


24 


27 
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Figure 7. Topics Covered in Science Curriculum Guides andTextbooks 



TIMSS Framework 


Population 1 


Population 2 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


1.1 Earth Sciences i 

1.1.1 Earth features 


1.1. 1.1 Composition 


















1.1. 1.2 Land forms 








/ 








/ 


1.1. 1.3 Bodies of water 


/ 






/ 








/ 


1 . 1 . 1 .4 Atmosphere 
















/ 


1.1. 1.5 Rocks, soil 










/ 


/ 






1.1. 1.4 Ice forms 
















/ 


1.1.2 Earth processes 


1. 1.2.1 Weather and climate 


/ 


/ 




/ 


/ 


/ 




/ 


1.1. 2. 2 Physical cycles 








/ 








/ 


1.1. 2.3 Building and breaking 


















1.1. 2.4 Earth’s history 
















/ 


1.1.3 Earth in the universe 


1.1. 3.1 Earth in the solar system 


/ 


/ 




/ 








/ 


1.1. 3.2 Planets in the solar system 








/ 








/ 


1.1. 3.3 Beyond the solar system 


















1.1. 3.4 Evolution of the universe 


















1 .2 Life Sciences 

1.2.1 Diversity, organization, and 
structure of living things 


1.2. 1.1 Plants, fungi 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


y 


1.2.12 Animals 


/ 


/ 


/ 


/ 


/ 


y 




y 


1.2. 1.3 Other organisms 








/ 


/ 






y 


1.2. 1.4 Organs, tissues 


/ 


/ 


/ 


/ 


/ 


y 


y 


y 


1.2. 1.5 Cells 








/ 


/ 






y 
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Figure 7. Topics Covered in Science Curriculum Guides and Textbooks (continued) 



TIMSS Framework 


Popula 


tion 1 


Population 2 j 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


1.2.2 Life processes and systems 
enabling life functions 


















1.2. 2.1 Energy handling 








/ 


/ 


/ 




/ 


1.2. 2. 2 Sensing and responding 








V 


/ 


/ 




/ 


1.2. 2. 8 Biochemical 

processes in cells 


















1.2.3 Life spirals, genetic 

continuity, and diversity 


















1.2.8. 1 Life cycles 








/ 


/ 


/ 




/ 


1.2. 8. 2 Reproduction 








/ 


/ 


/ 




/ 


1.2. 8. 8 Variation and inheritance 


















1.2. 8. 4 Evolution, speciation, 
and diversity 








/ 


/ 






/ 


1.2. 8. 5 Biochemistry of genetics 


















1.2.4 Interactions of living things 


















1.2. 4.1 Biomes and ecosystems 






/ 


/ 


/ 






/ 


1.2. 4. 2 Habitats and niches 








/ 


/ 








1.2. 4. 8 Interdependence of life 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.2.4. 4 Animal behavior 








/ 


/ 








1.2.5 Human biology and health 


/ 








/ 








1.2. 5.1 Nutrition 








/ 








/ 


1.2. 5. 2 Disease 










/ 


/ 




/ 


1.3 Physical Sciences 


1.3.1 Matter 


















1.8. 1.1 Classification of matter 








/ 


/ 


/ 




/ 


1.8. 1.2 Physical properties 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.8. 1.8 Chemical properties 








/ 


/ 


/ 


/ 


/ 
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Figure 7. Topics Covered in Science Curriculum Guides and Textbooks (continued) 



TIMSS Framework 


Population 1 






Population 2 




of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


1.3.2 Structure of matter 


















1.3. 2.1 Atoms, ions, and molecules 








/ 


/ 


/ 




/ 


1.3. 2. 2 Macromolecules, crystals 


















1.3. 2. 3 Subatomic particles 










/ 






/ 


1 .3.3 Energy and physical properties 


















1.3. 3.1 Energy types, 

sources, conversions 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


1.3. 3. 2 Heat and temperature 










/ 


/ 




/ 


1.3. 3. 3 Wave phenomena 


















1.3. 3. 4 Sound and vibration 








/ 








/ 


1.3. 3. 5 Light 






/ 


/ 


/ 


/ 






1.3. 3. 6 Electricity 






/ 


/ 


/ 


/ 


/ 


/ 


1.3. 3. 7 Magnetism 










/ 








1.3.4 Physical transformations 


















1.3. 4.1 Physical changes 












/ 




/ 


1.3. 4. 2 Explanations of 
physical changes 


















1.3. 4. 3 Kinetic theory 


















1.3. 4. 4 Quantum theory and 
fundamental particles 


















1.3.5 Chemical transformations 


















1. 3.5.1 Chemical changes 
















/ 


1.3.5. 2 Explanations of 
chemical changes 










/ 


/ 






1.3.5. 3 Rate of change 
and equilibria 


















1.3.5. 4 Energy and 

chemical change 
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Figure 7. Topics Covered in Science Curriculum Guides and Textbooks (continued) 



TIMSS Framework 


Population 1 


Population 2 | 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


1.3. 5.5 Organic and 

biochemical changes 


















1.3. 5. 6 Nuclear chemistry 
















/ 


1.3. 5. 7 Electrochemistry 


















1 .3.6 Forces and motion 


















1.3.6. 1 Types of forces 








/ 


/ 


/ 




/ 


1.3. 6. 2 Time, space, and motion 












/ 


/ 


/ 


1.3. 6. 3 Dynamics of motion 


















1.3. 6. 4 Relativity theory 


















1.3. 6. 5 Fluid behavior 


















1 .4 Science, technology, 
and mathematics 


1 .4.1 Nature or conceptions 
of technology 
















/ 


1 .4.2 Interactions of science, 

mathematics, and technology 


















1.4. 2.1 Influence of mathematics 
and technology on science 


















1.4.2. 2 Applications of science in 
mathematics and technology 








/ 


/ 


/ 




/ 


1.4.3 Interactions of science, 
technology, and society 


















1. 4.3.1 Influence of science 

and technology on society 








/ 


/ 


/ 




/ 


1.4. 3. 2 Influence of society on 
science and technology 


















1.5 History of science 
and technology 


/ 




/ 
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Figure 7. Topics Covered in Science Curriculum Guides andTextbooks (continued) 



TIMSS Framework 


Population 1 


Population 2 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 


Currie. = Curriculum 


Currie. Text- 

Guides books 


Currie. Text- 

1 Guides books 


Currie. Text- 

Guides books | 


Currie. Text- 

Guides books | 


1.6 Environmental and 
resource issues related 
to science 




1.6.1 Pollution 


/ 








/ 


/ 


/ 


/ 


1 .6.2 Conservation of land, 

water, and sea resources 


/ 


/ 




/ 


/ 


/ 




/ 


1.6.3 Conservation of material 
and energy resources 


/ 






/ 


/ 


/ 




/ 


1 .6.4 World population 


/ 








/ 








1 .6.5 Food production and storage 


/ 






/ 


/ 






/ 


1 .6.6 Effects of natural disasters 










/ 






/ 


1 .7 Nature of science 


1 .7.1 Nature of scientific 
knowledge 








/ 


/ 




/ 


/ 


1 .7.2 The scientific enterprise 








/ 








/ 


1.8 Science and other 
disciplines 




1.8.1 Science and mathematics 








/ 








/ 


1 .8.2 Science and other 
disciplines 








/ 








/ 


Total number of topics 


15 


9 


9 


40 


39 


29 


10 


50 
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What do toe expect students to do xoith the content toe plan to cover? 

Figure 8 below indicates the performance expectations from the TIMSS 
mathematics framework included in the curriculum guides and textbooks in 
the United States and a composite of other participating countries. A 
means that the performance expectation was listed in at least 70 percent of 
the curriculum guides or textbooks. 



Figure 8. Performace Expectations Included in Mathematics Curriculum Guides and Textbooks 



TIMSS Framework 


Population 1 


Population 2 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 










Non-Algebra 


Algebra 


Currie. = Curriculum 


Currie. Text* 
Guides books 


Currie. Text- 
Guides books 


Currie. Text- 
Guides books 


Currie. Text- 
Guides books 


Currie. Text- 
Guides books 



2.1.1 Representing 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.1.2 Recognizing equivalents 


/ 


/ 




/ 


/ 


/ 




/ 




/ 


2.1.3 Recalling mathematical 
objects and properties 


/ 


/ 


/ 


/ 


/ 


/ 




/ 




/ 


2.2 Using routine procedures 


2.2.1 Using equipment 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.2.2 Performing routine 
procedures 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.2.3 Using more complex 
procedures 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.3 Investigating and 
problem solving 


2.3.1 Formulating and clarifying 
problems and situations 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.3.2 Developing strategies 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.3.3 Solving 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 



O BEST COPY AVAILABLE 

ERLC 

id ebook to examine school curricula 




43 I 











ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



Figure 8. Performace Expectations Included in Mathematics Curriculum Guides andTextbooks (continued) 



TIMSS Framework 


Population 1 


Population 2 


of Content Categories 
and Subcategories 


International 

Composite 


United 

States 


International 

Composite 


United 

States 














Non-Algebra 


Algebra 


Currie. = Curriculum 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


Currie. 

Guides 


Text- 

books 


2.3.4 Predicting 


/ 




/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.3.5 Verifying 


/ 




/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.4 Mathematical reasoning 


2.4.1 Developing notation 
and vocabulary 


/ 






/ 


/ 




/ 


/ 


/ 




2.4.2 Developing algorithms 


/ 






/ 








/ 




/ 


2.4.3 Generalizing 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.4.4 Conjecturing 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.4.5 Justifying and proving 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.4.6 Axiomatizing 






















2.5 Communication 


2.5.1 Using vocabulary 
and notation 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.5.2 Relating representations 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.5.3 Describing and discussing 






/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


2.5.4 Critiquing 








/ 








/ 




/ 


Total number of topics 


13 


9 


16 


20 


18 


17 


16 


20 


16 


19 



49 
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Figure 9. Percent of Textbook Space Devoted to Major Population 2 Science Performace Expectations 





Understanding 


Theorizing, 
Analyzing, 
and Solving 
Problems 


Using Tools, 
Routine 
Procedures, 
and Science 
Processes 


Investigating 
the Natural 
World 


Communicating 


Argentina 


71.0 


11.0 


12.0 


0.0 


1.0 


Australia 


79.0 


7.0 


8.0 


7.0 


7.0 


Austria 


80.0 


8.0 


2.0 


7.0 


2.0 


Belgium (FI) 


47.0 


11.0 


34.0 


10.0 


5.0 


Belgium (Fr) 


15.0 


3.0 


9.0 


10.0 


6.0 


Bulgaria 


48.0 


15.0 


4.0 


0.0 


3.0 


Canada 


71.0 


8.0 


17.0 


11.0 


4.0 


China, People’s Rep. of 


73.0 


13.0 


6.0 


2.0 


0.0 


Colombia 


98.0 


3.0 


0.0 


0.0 


0.0 


Cyprus 


70.0 


12.0 


10.0 


8.0 


0.0 


Czech Republic 


95.0 


6.0 


6.0 


1.0 


1.0 


Denmark 


73.0 


3.0 


18.0 


10.0 


0.0 


Dominican Republic 


29.0 


16.0 


2.0 


19.0 


1.0 


France 


68.0 


16.0 


16.0 


1.0 


1.0 


Germany 


55.0 


15.0 


30.0 


0.0 


0.0 


Greece 


90.0 


6.0 


1.0 


0.0 


0.0 


Hong Kong 


74.0 


2.0 


23.0 


0.0 


0.0 


Hungary 


87.0 


2.0 


8.0 


2.0 


0.0 


Iceland 


94.0 


2.0 


4.0 


2.0 


6.0 


Iran 


100.0 


0.0 


0.0 


0.0 


0.0 


Ireland 


85.0 


2.0 


4.0 


6.0 


1.0 


Israel 


77.0 


3.0 


10.0 


1.0 


0.0 


Italy 


85.0 


8.0 


5.0 


3.0 


1.0 


.Japan 


82.0 


7.0 


13.0 


10.0 


2.0 


Korea75.0 


19.0 


8.0 


2.0 


0.0 




Latvia 


52.0 


4.0 


13.0 


7.0 


0.0 


Lithuania 


87.0 


12.0 


6.0 


1.0 


1.0 


Mexico 


92.0 


3.0 


9.0 


2.0 


3.0 


Netherlands 


58.0 


16.0 


10.0 


4.0 


1.0 


New Zealand 


56.0 


4.0 


26.0 


21.0 


8.0 


Norway 


72.0 


4.0 


5.0 


14.0 


3.0 


Portugal 


89.0 


9.0 


13.0 


2.0 


2.0 


Romania 


83.0 


9.0 


6.0 


1.0 


0.0 


Russian Federation 


88.0 


15.0 


7.0 


1.0 


1.0 


Scodand 


90.0 


0.0 


2.0 


1.0 


7.0 


Singapore 


76.0 


3.0 


3.0 


19.0 


0.0 


Slovak Republic 


86.0 


12.0 


9.0 


2.0 


1.0 


Slovenia 


85.0 


14.0 


5.0 


2.0 


0.0 


South Africa 


86.0 


1.0 


14.0 


8.0 


0.0 


Spain 


62.0 


19.0 


4.0 


5.0 


3.0 


Sweden 


72.0 


7.0 


2.0 


8.0 


3.0 


Switzerland 


79.0 


7.0 


1.0 


5.0 


1.0 


Tunisia 


93.0 


3.0 


1.0 


2.0 


0.0 


United States 


88.0 


4.0 


7.0 


2.0 


1.0 
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Teacher Survey 

TIMSS researchers also surveyed seventh- and eighth-grade teachers in 
Japan, Germany, and the United States regarding the topics they actually 
cover in class, how much time they devote to each (using the TIMSS frame- 
works as references), their beliefs about pedagogical strategies, and the hours 
spent teaching each week. An example of the kind of question the survey 
answers is, “What are the topics most commonly taught?” Figure 10 presents 
the 10 topics most commonly taught in eighth-grade mathematics and 
science in the United States, Japan, and Germany, as reported by eighth- 
grade teachers. 



Figure 1 0. The 1 0 Most Commonly Taught Topics in Eighth-Grade Mathematics and Science 



United States 


Japan 


Germany 


Mathematics 

Other Numbers and 


Geometry: Congruence 


Equations and Formulas 


Number Concepts 


and Similarity 


Perimeter, Area, and Volume 


Number Theory and Counting 


Data Representation and Analysis 


Two-Dimensional Geometry 


Perimeter, Area, and Volume 


Patterns, Relations, and Functions 


Three-Dimensional Geometry 


Estimation and Number Sense 


Two-Dimensional Geometry 


and Vectors 


Percentages 


Proportionality Problems 


Measurement: Units 


Two-Dimensional Geometry 


Other Content 


Geometry: Congruence and 


Proportionality Concepts 


Estimation and Number Sense 


Similarity 


Proportionality Problems 


Proportionality Concepts 


Proportionality Problems 


Properties of Common and 


Measurement: Estimation 


Percentages 


Decimal Fractions 


and Error 


Patterns, Relations, 


Relationships of Common 


Equations and Formulas 


and Functions 


and Decimal Fractions 




Other Numbers and Number 






Concepts 


Science 

Nature of Science 


Human Diversity 


Environmental and Resource 


Structure of Matter 


Human Life Processes and Systems 


Issues 


Matter 


Chemical Transformations 


Nature of Science 


Science and Technology 


Structure of Matter 


Energy and Physical Processes 


Mathematics 


Life Processes and Systems 


History of Science 


Physical Transformations 


Matter 


and Technology 


Environmental and 


Energy and Physical Processes 


Heat, Temperature, Wave, 


Resource Issues 


Earth Processes 


and Sound 


Energy and Physical Processes 


Atmosphere 


Interactions of Living Things 


Chemical Transformations 


Environmental and 


Structure of Matter 


Heat, Temperature, Wave, 


Resource Issues 


Diversity, Organization, and 


and Sound 




Structure of Living Things 


History of Science 




Life Processes and Systems 


and Technology 
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Other Curriculum-Analysis Methods 

A. National Science Foundation (NSF) Review of Instructional Materi- 
als for Middle School Science and Framework for Review: Instruc- 
tional Materials for Middle School Mathematics 

B. American Association for the Advancement of Science (AAAS) 
Project 2061 Curriculum-Analysis Procedure 

C. California Department of Education Curriculum Frameworks 
and Instructional Resources: Mathematics Instructional Materials 
Evaluation Instrument and Rating Form 

D. Council of Chief State School Officers (CCSSO) State Curriculum 
Frameworks and Standards Map: Definitions of Categories and 
Concepts 
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National Science Foundation 
Review of Instructional Materials 
for Middle School Science 



In 1996, the National Science Foundation (NSF) undertook a review of 
comprehensive curriculum projects in middle-school science. As work on 
the study progressed, it became clear that the framework for review devel- 
oped to examine middle-school materials and the results of the panel’s 
findings might be helpful beyond NSF and that they could be useful to those 
in the field working to improve science education in schools, districts, and 
states. The purpose of this paper is to make such information available to 
this broader audience. 

Introduction 

Unlike earlier school reforms, current reforms focus on identifying what 
all students should know and be able to do. Efforts such as NSF’s Systemic 
Initiatives aim to create bold new visions of curriculum, assessment, and 
pedagogy to improve education for all children. The frameworks for these 
reforms are often found in national standards, such as those from the National 
Council of Teachers of Mathematics (NCTM) and the National Academy of 
Sciences’ National Science Education Standards (NSES). These national frame- 
works were largely grassroots efforts with contributions from teachers, parents, 
school administrators, and scientists and mathematicians. They provide 
consensus views on what content is most important to teach, suggestions for 
teachers about effective instructional strategies, suggestions for how to assess 
student learning, and, in the case of the NSES- , suggestions for implementa- 
tion. In addition, the American Association for the Advancement of Science 
(AAAS) has developed Benchmarks for Science Literacy , a compendium of spe- 
cific science literacy goals developed by scientists and educators that states, 
districts, and schools can use as a guide for a science curriculum. 

Many states and districts have developed curriculum frameworks in 
mathematics and science that build on or adapt these standards efforts. The 
question now is, Do we have the tools required to successfully transverse the 
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current educational terrain? The national and state frameworks perhaps set 
the compass and provide a large-scale map, but it falls on districts, schools, 
and teachers to identify the best materials and programs to make reform a 
reality. Without quality instructional materials, even the best teachers can 
make little headway. 

To investigate the current status of instructional materials, NSF con- 
ducted a review of its portfolio of comprehensive curricula for middle-school 
science (grades 5 to 9) in early 1996. The justifications for starting with 
middle-level science included the following: 

■ Earlier NSF-funded projects had resulted in several sets of compre- 
hensive materials at the elementary level. 

■ There were questions both in the field and at NSF about the avail- 
ability of quality comprehensive materials for middle-school stu- 
dents. 

■ There were several sets of middle-school materials at or near 
completion and, therefore, ready for review. 

The purpose of the Middle School Science Study was to answer the 
following questions: 

1. What are the characteristics of the portfolio of comprehensive 
instructional materials for middle-school science developed with 
NSF funds? 

2. How sufficiently do these materials provide for a comprehensive 
program for middle-school science consistent with the national 
standards for science education? 

The study included a review of comprehensive curricula, those that 
equal a year or more of course material, produced during the past decade or 
that were currently under development. The central criteria used in review- 
ing the instructional materials were as follows: 

1. Is the science content correct? 

2. How well do the materials provide for conceptual growth in science? 

3. How well do the materials align with the National Science Education 
Standards ? 
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Rationale and Background — Overview and History of NSF's Instructional 
Materials Development Program (IMD) 

It is NSF’s goal to “achieve excellence in U.S. science, mathematics, 
engineering, and technology education at all levels .” 1 One of the strategies 
for meeting this goal is to fund development of high-quality instructional 
materials with potendal for national impact that are consistent with state and 
national standards. NSF, through the IMD Program, supports the develop- 
ment of new comprehensive materials and new instrucdonal units and the 
revision of existing high-quality materials. 

Developing high-quality instructional materials is an expensive and 
long-term process, requiring contributions from numerous teachers, scien- 
tists, and mathematicians to ensure that the content and pedagogy are 
current and correct. Materials should contain activities that are engaging for 
and relevant to students and should provide sufficient guidance for teachers 
so that they can successfully implement them in their classrooms. Materials 
must provide for extensive pilot and field testing with diverse student popu- 
lations, and this often means dme-consuming revisions. Materials supported 
by NSF are often under development for five years or longer before they are 
ready for publication. High-quality instructional materials are a critical 
component of the reform effort. Reform is not possible without materials 
that contain cutting-edge science; provide for students’ conceptual growth 
over time; and contain engaging reading, experiments, and opportunities for 
teacher-directed student inquiry. 

In sum, the IMD Program seeks projects that: 

■ Involve collaboration of scientists, mathematicians, teachers, and 
educators; 

■ Apply current research in teaching and learning; 

■ Align with standards; 

■ Contain embedded student assessments that help inform instruc- 
tion and use a variety of strategies to assess student learning; 

■ Field-test materials in diverse settings; and 

■ Employ formative and summative evaluations that include student 
outcome data from field-test sites . 2 
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Overview of Science Materials Reform 

For mathematics, NSF funded a portfolio of projects to develop compre- 
hensive instructional materials following the release of the National Council 
of Teachers of Mathematics Standards for Content and Evaluation in 1989 3 ; 
however, NSF began funding the development of innovative comprehensive 
instructional materials in science a decade prior to the release of the science 
standards. The American Association for the Advancement of Science 
(AAAS) released Benchmarks for Science Literacy 4 in 1993, and the National 
Academy of Sciences/National Research Council published the National 
Science Education Standards (NSESf in December 1995. Therefore, many of 
the middle-school projects reviewed in this study predate these standards. 
Many curriculum developers, however, served on the working groups that 
developed the Benchmarks and standards, participated in the extensive review 
and critique of the science standards, and incorporated ideas emerging from 
these standards-based projects in their materials. 

The current cycle of development of instructional science materials, 
dating from the mid-1980s, is driven by (1) the need to ensure that there are 
effective materials available, particularly at the elementary level, where sci- 
ence is frequently relegated to “the last 20 minutes on Friday afternoon”; (2) 
the need to develop materials that provide more “hands-on” opportunities in 
which students can actively conduct their own observations and experiments 
and generate their own questions under teacher guidance; and (3) the need 
to incorporate new research findings in teaching and learning into science 
instructional materials. 

In the mid-1980s, NSF funded hands-on materials at the elementary 
level, through the TRIAD projects. These projects formed partnerships 
among three critical groups — publishers, developers, and school districts — 
and operated on the assumption that the availability of high-quality elemen- 
tary science materials would motivate teachers to teach science using a 
hands-on and inquiry-based approach. Shortly thereafter, work began on 
developing middle school materials, and this was followed by work on high 
school materials, many of which are now nearing completion. 6 

NSF has refined its ideas about what constitutes good instructional 
materials. NSF increasingly is concerned that materials provide appropriate 
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guidance for teachers, suggest appropriate instructional strategies, contain a 
variety of assessment activities, accommodate the diversity of students, and 
contain suggestions for implementation. The review instrument developed 
by Inverness Research and modified by NSF for use in the Middle School 
Study reflects the Foundations concern with assessing a wide range of 
features in addition to a high-quality content. 

Procedures of the Middle School Science Study 

The Middle School Science Study adapted NSF’s peer review process 
to critique the portfolio of comprehensive curricula, using a method similar 
to that used by the IMD program for reviewing proposals for new projects. 
In this peer review process, panels of outstanding scientists, mathemati- 
cians, and educators critique proposals submitted for funding and make 
recommendations to the NSF about each proposals quality, funding prior- 
ity, and potential impact. Typically, reviewers provide individually written 
reviews, discuss the proposals in panel meetings, and develop a panel 
summary for each proposal. This study followed a four-step process in 
completing the review of materials: (1) training, (2) independent review, (3) 
summary and consensus, and (4) synthesis. 



A review panel of 20 experts comprising scientists, science and technol- 
ogy educators, and science teachers participated in the peer review process. 
For the peer review, program directors from the Division of Elementary, 
Secondary, and Informal Education (ESIE) met with the panel of experts to 
agree on the process and criteria for reviewing the materials. The panel used 
an instrument developed by Inverness Research to review one instructional 
module as a trial run in calibrating the review process. Following the trial 
review, the panel critiqued and revised the instrument to develop a common 
understanding for each item and to agree on the review process. Appendix A 
on page 61 includes a copy of the final review instrument, called the Frame- 
work for Review. 

Independent Review 

Following the panel meeting, panel members formed small working 
groups, comprising a scientist, science educator, practitioner, and individuals 
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with expertise in assessment and implementation. Panel members read their 
assigned portions of the curriculum materials at home and prepared in- 
depth analyses of the materials using the review instrument as a framework 
for guiding their cridques. The panel members were asked to provide de- 
tailed jusdfications for their radngs for each item of the instrument. 

Panel Summary 

The panel members returned to NSF and exchanged results in their 
working groups. Each group prepared a written summary for each program 
representing a consensus of their reviews. The panel provided feedback on 
the review instrument and the review process, which NSF staff used to revise 
the instrument for future use. New summary groups were formed to discuss 
cross-cutting issues: (1) treatment of science content, (2) approach to teach- 
ing, (3) approach to assessment and equity, and (4) strategies for implemen- 
tation. Each summary group reported to the whole panel and, through a 
large-group discussion, developed the major summary findings of the overall 
peer review. 



Synthesis 

A second panel of experts convened to review the process and findings 
of the peer review, to develop strategies for disseminating the findings, and 
to recommend future directions. The synthesis panel constituted 14 mem- 
bers, including four from the peer review panel. The synthesis panel in- 
cluded scientists, teachers, curriculum developers, and national and state 
leaders in the reform of science, mathematics, and technology education. 
The synthesis panel carefully reviewed the panel summaries and summary 
recommendadons from the peer review process and developed an overall 
synthesis of findings that are the basis for this report. 



Constraints 

The review procedures were designed to provide a broad-brush assess- 
ment of the status of the portfolio of NSF-funded comprehensive instruc- 
tional materials for middle school science. The purpose of the activity was to 
identify strengths and weaknesses in the portfolio, as well as gaps requiring 
the development or revision of projects. The intent of the study was to 
provide feedback to program officers who review proposals. 
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The study was not designed to (1) provide the NSF vision of a national 
curriculum, (2) thoroughly evaluate the individual projects, (3) offer a “con- 
sumer report” on quality of curricula, or (4) survey the needs of teachers and 
schools. 

The study had several constraints: 

■ In most cases, the complete set of materials for one comprehensive 
program was not reviewed by all members of the panel. Therefore, 
each panel member completed the individual review based on only a 
subset of the full package of materials. 

■ No materials were reviewed by more than one panel; thus, it was not 
possible to equate a particular value on an item for one set of mate- 
rials given by one panel with a value for the same item given by a 
different panel to another set of materials. 

■ Panel members analyzed, in general terms, the degree to which a set 
of materials addressed content standards within particular science 
disciplines, but did not do a fine-grained analysis of specific con- 
cepts and the amount of time allocated to the mastery of those 
specific concepts. 

While the results of the study have shed light on the current status of 
middle-school science instructional materials developed with NSF funding, 
they do not serve as a detailed evaluation of the individual projects. It is 
hoped that the results of the study will be used to inform state and local 
administrators, curriculum developers, principal investigators of systemic 
reform and teacher enhancement projects, and NSF program officers about 
quality, standards-based instructional materials for middle school science. 

The review instrument developed as part of this study is an important 
product for use by those who select materials or school science programs. 

Results — Overall 

Thirteen of the 19 projects examined as part of NSF s Middle School 
Review had panel ratings of 3 or higher on a 1-5 point scale of the Inverness 
Research Frameiuork for Revieiu, with 1 as “low” and 5 as “high” on overall 
quality. Eight of these — Prime Science (6-10); Science 2000 (6-8); Science and 
Technology : Investigating Human Dimensions (BSCS, 6-8); Full Option Science 



.ERIC 

i ^“ a ^,UIDEB00K TO EXAMINE SCHOOL CURRICULA 




55 1 



ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



System ( FOSS 5-6); Science and Technology for Children (5-6); Improving Urban 
Elementary Science ( Insights 5-6); Elementary School Science and Health Materials 
( BSCS 5-6); and Integrated Mathematics , Science , and Technology (IMaST 7-8) — 
are multiyear comprehensive programs. Event-Based Science and Junior High/ 
Middle School Life Science Program (Jeffco) comprise materials for one year. 
Science Education for Public Understanding Program (SEPUP) and its predeces- 
sor, Chemical Education for Public Understanding Project (CEPUP), cover non- 
sequential muldple single years of material, and a third set of materials, Life 
Science for Public Understanding Project , is currently under development in 
order to complete a comprehensive grade 7-10 series. National Geographic 
Kids Network (4-6) covers multiple grade levels but is not designed to cover a 
full year of science at any grade level. Therefore, the answer to the question 
regarding the availability of high-quality, standards-based middle-school 
science materials is that there are some good, comprehensive programs. 

Also, one-year programs can form important components of a total middle- 
school program. (See Appendix B for brief descripdons of these projects on 
page 97.) 



Science content in middle schools includes important scientific con- 
cepts in earth, biological, and physical sciences; opportunities for inquiry; 
and information on the history and processes of science. Particular pro- 
grams stand out as having strengths in particular areas. These programs 
have the potential to serve as exemplars for curriculum developers who are 
designing new materials and for school districts that are forming school 
science programs. 

Projects vary in their approach to content. A few developers have 
produced multiyear comprehensive programs designed to achieve all of the 
content standards for the middle level. These programs have been forced to 
face the challenge of finding the best balance between breadth and depth of 
science content. One example, PRIME Science , provides a balanced curricu- 
lum covering biological, earth, and physical sciences for grades 6 to 10 and 
revisits important concepts so students can deepen their understanding of 
key ideas. Ajiother example, Science 2000 , is rated high for its alignment with 
the National Science Education Standards content standards and its develop- 
ment of key science concepts. Science 2000 is unique in that it is organized 
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around a few major conceptual themes, and it has separate units on science 
and technology. Integrated Mathematics, Science, and Technology (IMaST) stands 
alone as a program that is designed to integrate the teaching of science and 
mathematics with technology. IMaST is designed to be taught by a team of 
mathematics, science, and technology teachers in a three-hour block and 
enables students to achieve the content standards in science and mathematics 
for the middle grades, with a grounding in technology education as well. 

Single-year programs for the middle grades do not propose to meet all 
of the content standards for grades 5 to 9. Programs of this type either have 
developed materials aligned with a discipline-based approach (e.g., Junior 
High/Middle School Life Science, or Event-Based Science: Earth Science) or they 
have taken a problem/issue-centered approach that may transcend science 
disciplines ( CEPUP and SEPUP). Event-Based Science: Earth Science takes 
both approaches in that it is a problem-centered program designed to teach 
the traditional content of earth science. Event-Based Science is rated high in 
its presentation of science content, and it is one of the few programs that 
addresses earth science. It effectively uses video footage of natural disasters 
(e.g., earthquakes, tornadoes, floods) to engage students in investigating the 
content and processes of earth science. The strength of the single-year 
approach is that a school district can build its own multiyear program by 
selecting single-year programs that fit their curriculum framework. 

Comprehensive programs for grades 5 to 6 that are part of a K-6 pro- 
gram are challenged with the need to be both scientifically meaningful and 
developmentally appropriate for young students. The Full Option Science 
System (FOSS) for grades 5 to 6 is an example of the effective treatment of 
science content at this level. FOSS strikes a good balance between an empha- 
sis on the major conceptual themes, such as systems, and an emphasis on 
science concepts, such as an electrical circuit. The reviewers felt that FOSS 
presents important current science content accurately, at a developmentally 
appropriate level, and covers appropriate breadth of science and depth of 
understanding. 



Pedagogy 



Good materials contain suggestions for teachers, such as sequencing 
activities to achieve desired learning results, and hints on working with groups 
of students. Particular programs have the potential to serve as exemplars for 
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particular areas related to pedagogy. From the category of multiyear compre- 
hensive programs, PRIME Science was rated highly by the panel for overall 
quality of pedagogical design. The panel members were especially impressed 
with the manner in which PRIME Science presents a logical progression of the 
development of conceptual understanding that reflects researchers’ current 
understanding of the teaching and learning of science. Science and Technology 
far Children is a K-6 program that received high marks for engaging students in 
science inquiry and technology problem solving. CEPUP was recognized as 
providing a good model for using personal and social issues as the pedagogical 
driver for engaging students in learning and applying important science 
concepts. National Geographic Kids Network is unique in its effective use of 
telecommunications for engaging students in collaborative science investiga- 
tions, and Science 2000 provides a model for using interactive videodisc tech- 
nology to engage students in learning science content. 



Assessment 

While classroom assessment is an important component of instructional 
materials, some of the materials (particularly those funded before the early 
1990s) contained limited assessment activities. Other projects appear to 
include assessment as an afterthought. It is now believed that assessments 
should be developed concurrently with, and embedded in, the instructional 
materials. Some materials, such as Junior High/Middle School Life Science 
Program (Jeffco), contain traditional assessments (paper-and-pencil tests), but 
they are well done. Others have greatly expanded on this traditional base by 
including assessments in which students demonstrate through performance 
or extended response questions what they know and are able to do. Event- 
Based Science, for example, is regarded as being very user-friendly for teach- 
ers and has excellent scoring rubrics that are related to the ongoing instruc- 
tional themes. At the elementary level, FOSS, Insights, and Science and Tech- 
nology for Children include embedded assessments that are integral to instruc- 
tion and use a variety of approaches to test student understanding. SEPUP 
was cited as an outstanding example of embedded assessment at the middle- 
school level. National Geographic Kids Network also includes innovative uses of 
performance assessment linked to computer network communications. 
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Equity 

Panel members described the approaches to equity in these materials as 
more likely to commit sins of omission than commission. Many of the mate- 
rials simply do not address equity issues in any explicit way, although there is 
no obvious bias in the materials. Panel members felt that almost all of the 
materials would benefit from an explicit focus on equity issues and concrete 
suggestions for how teachers can gain access to needed materials and sup- 
plies, with an understanding that programs that rely on complex technolo- 
gies may be expensive and thereby excluded in many schools and districts at 
the current time. Supplemental materials are needed that address the effec- 
tive use of heterogeneous student groups and the importance of accommo- 
dating various learning styles. These materials may be produced by others 
than the curriculum developers. 

Panel members lauded materials that focused on societal issues, such as 
Event-Based Science , SEPUP , and Middle School Science and Technology ( BSCS 6- 
8), as having an inclusionary effect, because they address many issues using 
events and materials familiar and relevant to students. These integrated 
approaches provide access to important scientific ideas. Both sets of materials 
also discuss student learning styles and suggest cooperative learning strategies. 

Implementation 

Most of the materials packages in review do not address dissemination or 
implementation issues, and this is critical in focusing schools and districts on 
strategies for scaling up projects, exerting quality control, aligning curriculum, 
professional development and assessment, working effectively with parents and 
other community members, and so forth. Notable exceptions in this regard are 
the implementation guides including Middle School Science and Technology 
(BSCS 6-8) and Junior High/Middle School Life Science Program (Jeffco). Specific 
suggestions and strategies are provided for adopting new approaches to 
appropriate professional development, for scope and sequence of content, and 
for evaluating the effectiveness of implementing the materials. None of the 
materials packages mentions how to work with parents or the public. 
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Conclusions 

There are a number of high-quality middle-school science curricula 

available, and some are comprehensive. With care and these materials, 

schools and districts can create good middle-school science programs. 

A few key findings are: 

■ Most of the 13 sets of materials that rated 3 or higher on the 5-point 
scale are generally consistent with the nadonal science content 
standards. 

■ The emphasis on science literacy for all students points to the 
importance of applying equity principles, but most materials do not 
explicitly address strategies for improving the performance of a 
diverse set of students through attention to differences in ability, 
learning style, and so on. Additional supplemental materials may be 
needed to provide good strategies. 

■ Among the content areas, earth science appears least frequently. 

■ Connections between science and mathematics are not developed in 
most of the materials. 

■ The greatest weakness in the set of materials relative to the science 
standards is the lack of sufficient focus on the history and nature of 
science. 

■ Too few materials incorporate significant and appropriate use of 
instructional technologies, such as ensuring that materials are 
presented in a variety of formats. 
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APPENDIX A 

Framework for Review: 

Instructional Materials for Middle School Science 

Title: 

Author (s): 

Publisher: Copyright date: 

Reviewed by: Date: 

I. Descriptors 

a. Write a brief description of the components of the curriculum upon 
which this review is based (e.g., teacher’s guide, student books, hands- 
on materials, multimedia material). That is, what materials did you 
receive and include in your review? 



b. Write a brief description of the purpose and broad goals of these 

materials. That is, what were the stated purposes, and what were the 
actual results of the materials? 



c. What grade levels do the materials serve? 

5 6 7 8 
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d. Are the instructional materials designed to 

provide a complete muldyear program for middle school science. 

provide a complete one-year course for middle school science. 

provide muldple modules or units that could be used to supplement other course 

materials for middle school science. 

provide a single module or collection of acdvides that could be used to supplement 

other course materials for middle school science. 

other (explain): 



e. What are the major domains/ topics of the content covered by these materials? 



II. Quality of the Science 

Directions: For each item, circle the number corresponding with your response to the question . 

Write an explanation for your rating of each item below the item . 



a. Does the content in the instructional materials align well with all eight areas of the Content 
Standards as described in the National Science Education Standards (NSESJ? 

(See attached guidelines.) 



1 2 


3 


4 5 


Omits substantial content 


Some misalignment 


The curriculum 


included in NSES 


of content with 


aligns well with 


and/or includes substantial content 


recommendations 


content recommendations 


not recommended in NSES 


in NSES 


in NSES 
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b. Are the science concepts presented in the instructional materials accurate and correct? 

(Provide examples of major errors where they are evident. Attach extra page if necessary.) 



1 



2 



3 



4 



5 



Substantial, major errors 



Mostly correct, with 
some minor errors 



Scientifically accurate 
and correct 



c. Do the instructional materials adequately present the major concepts in the standards and 
adequately demonstrate and model the processes of science? 



1 



2 



3 



4 



5 



Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 



d. Does the science presented in the instructional materials reflect current disciplinary knowledge? 



1 



2 



3 



4 



5 




The ideas are 
out of date 



Somewhat current 



Current 
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e. Do the instructional materials accurately represent views of science as inquiry as described in 
the National Science Education Standards ? 

1 2 3 4 5 

Poor examples Mixed Rich and accurate 

of inquiry quality examples of inquiry 



f. Do the instructional materials accurately present the history of science? 

1 2 3 4 5 

Poor portrayal Mixed Rich and accurate 

of history of science quality portrayal of 

history of science 



g. Do the materials emphasize technology as an area of study? 



1 


2 


3 


4 


5 


Little or no 
emphasis 




Some emphasis 




Rich and well- 
designed emphasis 
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h. Do the materials emphasize the personal and societal dimensions of science? 



1 


2 


3 


4 


5 


Little or no 
emphasis 




Some emphasis 




Rich and well- 
designed emphasis 



i. Do the materials emphasize the content of life science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well- 

emphasis designed emphasis 



Do the materials emphasize 


the content of earth science? 






1 2 


3 


4 


5 


Little or no 


Some emphasis 




Rich and well- 


emphasis 






designed emphasis 
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k. Do the materials emphasize the content of physical science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well- 

emphasis designed emphasis 



1. Do the instructional materials provide sufficient activities for students to develop a good 
understanding of key science concepts? 



1 


2 


3 


4 5 


Too few learning 




Activities provide 


Activities provide 


activities 




some opportunity 


many rich opportunities 






for students to learn 


to learn key science 






some important concepts 


concepts 



m. Do the instructional materials provide sufficient opportunities for students to apply their 
understanding of the concepts (i.e., designing of solutions to problems or issues)? 

1 2 3 4 5 

Very few Some Very rich in 

application activities application activities application activities 
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n. Do the instructional materials present an accurate picture of the nature of science as a 
dynamic endeavor? 



1 



2 



3 



4 



5 



The image of science is 
out-of-date, inaccurate, 
or non-existent 



The image of 
science is of 
mixed quality 



The image of 
science is current 
and accurate 



o. Do the materials develop an appropriate breadth and depth of science content? 

1 2 3 4 5 

Too narrow Somewhat Good balance of 

or too broad balanced breadth and depth 



p. What is the overall quality of the science presented in the instructional materials? 



1 



2 



3 



4 



5 



Low 



Medium 



High 
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III. The Pedagogical Design 

a. Do the instructional materials provide a logical progression for developing conceptual 
understanding in science? 

1 2 3 4 5 

No logical Somewhat logical Logical progression 

progression progression of ideas of ideas that builds 

of ideas conceptual understanding 



b. Do the instructional materials provide students with the opportunity to make conjectures, 
gather evidence, and develop arguments to support, reject, and revise their preconceptions 
and explanations for natural phenomena? 



1 


2 


3 


4 


5 


No opportunity 




Some opportunity 




Rich and well- 
designed opportunity 



c. To what extent do the instructional materials engage students in doing science inquiry? 

1 2 3 4 5 

Many rich and authentic 
opportunities for students 
to do science inquiry 



o 
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activities for students for students to do 

to do science inquiry science inquiry 
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d. To what extent do the instructional materials engage students in doing technology problem 
solving? 



1 



2 



3 



4 



5 



Very few or very contrived 
activities for students 
to do technology 
problem solving 



Some good activities 
for students to do 
technology 
problem solving 



Many rich and authentic 
opportunities for students 
to do technology 
problem solving 



e. To what extent does the curriculum engage students in activities that help them connect 
science to everyday issues and events? 

1 2 3 4 5 

Very few or very contrived Some good activities Many rich and authentic 

activities for students for students to make opportunities for students 

to make connections connections to make connections 



f. How would you rate the overall developmental appropriateness of the instructional materi- 
als, given their intended audience of ALL students at the targeted level (s)? 



1 



2 



3 



5 



3 



Not developmental^ 
appropriate 



Somewhat developmen tally 
appropriate 



4 



Developmentally 

appropriate 
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Do the materials reflect current knowledge about effective teaching and learning practices 
(e.g., acdve learning, inquiry, community of learners) based on research related to science 
education? 

1 2 3 4 5 

Do not reflect current Somewhat reflective Reflect well current 

knowledge about of current knowledge knowledge about 

teaching and learning about teaching and learning teaching and learning 



h. Do the instructional materials provide students with the opportunity to clarify, refine, and 
consolidate their ideas, and to communicate them through multiple modes? 



1 


2 


3 


4 


5 


No opportunity 




Some opportunity 




Rich and well- 
designed opportunity 



i. Do the instructional materials provide students with the opportunity to think and communi- 



cate scientifically? 
1 


2 


3 


4 


5 


No opportunity 




Some opportunity 




Rich and well- 
designed opportunity 
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j. Do the instructional materials provide students with activities connecting science with other 



subject areas? 










1 


2 


3 


4 


5 


No opportunity 




Some opportunity 




Rich and well- 
designed opportunity 



k. Are the instructional materials likely to be interesting, engaging, and effective for students? 

12 3 45 

Not at all interesting Somewhat interesting Interesting and engaging 



1. Are the instructional materials likely to be interesting, engaging, and effective for girls and 
for boys? 



1 


2 


3 


4 


5 


Gender biased 




Some sensitivity 
to gender issues 




Equally interesting, 
engaging, and effective 
for girls and for boys 
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m. Are the instructional materials likely to be interesting, engaging, and effective for 

underrepresented and underserved students (e.g., ethnic, urban, rural, with disabilities)? 



1 


2 


3 


4 5 


Biased 




Some sensitivity 
to underrepresented and 
underserved students 


Equally interesting, 
engaging, and effective for 
underrepresented and 
underserved students 



n. Does assessment have explicit purposes connected with decisions to be made by teachers 
(e.g., prior knowledge, conceptual understanding, grades)? 



1 


2 


3 


4 


5 


Unclear purposes 




Somewhat clear 




Clear statement 




purposes 




of purposes 



o. Do assessments focus on the curriculum’s important content and skills? 



1 


2 


3 


4 


5 


Poor correspondence 




Fair correspondence 




Full correspondence 



ERIC 

. MMiMilH 

I 72 




GUIDEBOOK TO EXAMINE SCHOOL CURRICULA 



ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



p. Do the instructional materials include multiple kinds of assessments (e.g., performance, 
paper/pencil, portfolios, student interviews, embedded, projects)? 



1 


2 


3 


4 


5 


Little or no student 




Some variety of 




Complete student 


assessment provided 




student assessment 




assessment package 



q. Are the assessment practices fair to all students? 

1 2 3 4 5 

Fair to a few Fair to most Fair to all 



r. Do the instructional materials include adequate and appropriate uses of a variety of educa- 
tional technologies (e.g., video, computers, telecommunications)? 



1 


2 


3 


4 


5 


Little or no 




Some educational 




Many appropriate rich 


educational technology 




technology included 




and useful applications 


included 








of educational 










technology included 
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s. What is the overall quality of the pedagogical design of these instructional materials? 

1 2 3 4 5 

Low Medium High 



IV. Implementation and System Support 

a. Will the teachers find the materials interesting and engaging? 

1 2 3 4 5 

Dry and boring Somewhat interesting Interesting and engaging 

and engaging 



b. Do the instructional materials include information and guidance to assist the teacher in 
implementing the lessons? 



1 


2 


3 


4 


5 


No teacher support 




Some teacher support 




Rich and useful 
teacher support 
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c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the required science materials? 



1 


2 


3 


4 


5 


No materials support 




Some materials support 




Rich and useful 
materials support 



d. Do the instructional materials provide information about how to establish a safe science 



learning environment? 








1 2 


3 


4 


5 


No safety information 


Some safety information 




Rich and useful 
safety infonnation 



e. Do the instructional materials provide information about the kinds of professional develop- 
ment experience needed by teachers to implement the materials? 



1 2 3 4 5 

Little or no Partial information Rich and useful 

information provided provided information provided 



O 

_! RJC 

S3Z aSiUlDEBOOK TO EXAMINE SCHOOL CURRICULA 



T5T 



ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



f. Do the materials provide guidance in how to link the materials with the district and state 
assessment frameworks and programs? 



1 


2 


3 


4 5 


No guidance 




Some guidance 


Rich and useful guidance 



g. Do the materials provide guidance and assistance for actively involving administrators, 
parents, and the community-at-large in supporting school science? 



1 


2 


3 


4 5 


No guidance 




Some guidance 


Rich and useful guidance 



h. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 

1 2 3 4 5 

Teacher-unfriendly Somewhat teacher-friendly Teacher-friendly 
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V. Major Strengths and Weaknesses 

a. In your opinion, what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 



VI. Overall Quality, Value, and Contribution 



a. 


In your opinion, what is the overall quality of these materials reladve to: 










Low 








High 




turning students on to science? 


1 


2 


3 


4 


5 




making students think? 


1 


2 


3 


4 


5 




quality of science content? 


1 


2 


3 


4 


5 




quality of pedagogy? 


1 


2 


3 


4 


5 




quality of classroom assessments? 


1 


2 


3 


4 


5 




encouraging teachers to teach differendy? 


1 


2 


3 


4 


5 


b. 


In your opinion, what is the overall quality of these instrucdonal materials? 








1 2 


3 




4 







Low Medium High 



c. To what extent would you encourage the dissemination, adopdon, and implementadon of 
this curriculum? 



1 



2 



3 



4 



5 



Not worthy of 
dissemination, adoption, 
or implementation 
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: if revised 




OK to disseminate, 
adopt, and implement 
as is 
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Framework for Review: 

Instructional Materials for Middle School Mathematics 

Title: 

Author (s) : 

Publisher: Copyright date: 

Reviewed by: Date: 

I. Descriptors 

a. Write a brief description of the components of the curriculum upon which this review is 

based (e.g., teachers’ guide, student books, hands-on materials, multimedia material). That 
is, what materials did you receive and include in your review? 



b. Write a brief description of the purpose and broad goals of these materials. That is, what 
were the stated purposes, and what were the actual results of the materials? 



c. What grade levels do the materials serve? 

5 6 7 8 
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d. Are the instructional materials designed to 

provide a complete multiyear program for middle school mathematics? 

provide a complete one-year course for middle school mathematics? 

provide multiple modules or units that could be used to supplement other course 

materials for middle school mathematics? 

provide a single module or collection of activities that could be used to supplement 

other course materials for middle school mathematics? 

other (explain) : 



e. What are the major domains/ topics of the content covered by these materials? 
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II. Quality of the Mathematics 

Directions: For each item , circle the number corresponding with your response to the question . Write an 
explanation for your rating of each item below the item. 

a. Does the content in the instructional materials align well with all 13 areas of the Curriculum 
Standards as described in the National Council of Teachers of Mathematics’ (NCTM) 
Curriculum and Evaluation Standards for School Mathematics ? 

(See attached guidelines.) 

12 3 45 

Omits substantial content Some misalignment The curriculum 

included in NCTM of content with aligns well with 

and/ or includes substantial content recommendations content recommend- 

not recommended in NCTM in NCTM ations in NCTM 



b. Are the mathematics concepts presented in the instructional materials accurate and correct? 
(Provide examples of major errors where they are evident. Attach extra page if necessary.) 



2 



3 



4 



5 



Substantial, major errors 



Mosdy correct, with 
some minor errors 



Mathematically accurate 
and correct 
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c. Do the instructional materials adequately present the major concepts and adequately dem- 
onstrate and model the processes of mathematics? 

1 2 3 4 5 

Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 



d. Do the instructional materials accurately represent views of mathematical problem solving as 
described in the NCTM Curriculum and Evaluation Standards for School Mathematics} 



1 2 3 4 5 

Poor portrayal Mixed Rich and accurate 

of problem solving quality portrayal of 

problem solving 



e. Do the materials use technology as a tool for learning mathematics? 



1 


2 


3 


4 


5 


Little or no 
use 




Some emphasis 




Rich and well- 
designed use 
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f. Do the materials emphasize communication about mathematics through a variety of modali- 
ties? 



1 



2 



3 



4 



5 



Little or no 
emphasis, few 
modalities 



Some emphasis, Rich and well- 

some modalities designed emphasis, 

varied modalides 



g. Do the materials appropriately address mathematical reasoning? 

1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



h. Do the materials appropriately address computation? 

1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 
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i. Do the materials appropriately address estimation? 

1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



j. Do the materials appropriately address numbers and number relationships? 



1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



k. Do the materials appropriately address number systems and number theory? 



1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 
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1. Do the materials appropriately address patterns? 

1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



m. Do the materials appropriately address functions? 

1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



n. Do the materials appropriately address algebra? 

1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 
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o. Do the materials appropriately address geometry? 

12 3 45 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



p. Do the materials appropriately address measurement? 



1 2 3 45 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



q. Do the materials appropriately address statistics? 



12 3 45 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 
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r. Do the materials appropriately address probability? 

1 2 3 4 5 

Not appropriately Somewhat Appropriately 

addressed appropriately addressed addressed 



Do the instructional materials provide sufficient activities for students to develop a good 
understanding of key mathemadcs concepts? 

1 2 3 4 5 

Too few learning Activities provide some Activities provide many rich 

activities opportunity for students to learn opportunities to learn key 

some important concepts mathematics concepts 



t. Do the instructional materials provide sufficient opportunities for students to apply their 
understanding of the concepts (i.e., designing of solutions to problems or issues)? 



1 


2 


3 


4 


5 


Very few 

application activities 




Some 

application activities 




Very rich in 
application activities 
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u. Do the materials develop an appropriate breadth and depth of mathematics content? 



1 2 3 4 5 

Too narrow Somewhat Good balance of 

or too broad balanced breadth and depth 



v. What is the overall quality of the mathematics presented in the instructional materials? 

1 2 3 4 5 

L° w Medium High 



III. The Pedagogical Design 

a. Do the instructional materials provide a logical progression for developing conceptual 
understanding in mathematics? 



12 3 45 

Somewhat logical Logical progression 

progression of ideas of ideas that builds 

conceptual understanding 



No logical 
progression 
of ideas 
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b. Do the instructional materials provide students the opportunity to formulate, solve, and 
reflect critically on problems? 



1 


2 


3 


4 


5 


No opportunity 




Some opportunity 




Rich and well-designed 
opportunity 



c. To what extent are the mathematical concepts embedded in learner-appropriate contexts? 



1 2 

Very few or very 
contrived activities 
for students to do 
mathemadcal problem solving 



3 

Some good acdvides 
for students to do 
mathematical problem solving 



4 5 

Many rich and authentic 
opportunities for students 
to do mathematical 
problem solving 



d. How would you rate the overall developmental appropriateness of the instructional materi- 
als, given their intended audience of ALL students at the targeted level (s)? 

1 2 3 4 5 

Not developmentally Somewhat developmentally Developmentally 

appropriate appropriate appropriate 
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e. Do the materials reflect current knowledge (that is, in the last 5 years) about effective 

teaching and learning practices (e.g., active learning, inquiry, community of learners) based 
on research related to mathematics education? 

1 2 3 45 

Do not reflect current Somewhat reflective Reflect well current 

knowledge about of current knowledge knowledge about 

teaching and learning about teaching and learning teaching and learning 



Do the instructional materials provide students the opportunity to clarify, refine, and con- 



solidate their ideas? 








1 2 


3 


4 


5 


No opportunity 


Some opportunity 




Rich and well- 
designed opportunity 



g. Do the instructional materials provide students with activities connecting mathematics with 



other subject areas? 








1 2 


3 


4 


5 


No opportunity 


Some opportunity 




Rich and well 
designed opportunity 
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h. Are the instructional materials likely to be interesting, engaging, and effective for girls and 
for boys? 



1 


2 


3 


4 


5 


No sensitivity 
to gender issues 




Some sensitivity 
to gender issues 




Sensitive to 
gender issues 



Are the instructional materials likely to be interesting, engaging, and effective for 
underrepresented and underserved students (e.g., ethnic, urban, rural, with disabilities)? 



1 



2 



3 



4 



5 



No sensitivity 
to underrepresented and 
underserved students 



Some sensitivity 
to underrepresented and 
underserved students 



Sensitive to 
underrepresented and 
underserved students 



Does assessment have explicit purposes connected with decisions to be made by teachers 
(e.g., prior knowledge, conceptual understanding, grades)? 



1 


2 


3 


4 


5 


Unclear purposes 




Somewhat clear 
purposes 




Clear statement 
of purposes 



ERIC 

~T~U2ILSiB&lSMJ ' 

190 



37 



GUIDEBOOK TO EXAMINE SCHOOL CURRICULA 



ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



k. Do assessments focus on the curriculum’s important content and skills? 



1 


2 


3 


4 


5 


Poor correspondence 




Fair correspondence 




Full correspondence 



1. Do the instructional materials include multiple kinds of assessments (e.g., performance, 
paper/pencil, portfolios, student interviews, embedded, projects)? 



1 


2 


3 


4 


5 


Little or no student 




Some variety of 




Complete student 


assessment provided 




student assessment 




assessment package 



m. Are the assessment practices fair to all students? 

1 2 3 4 5 

Fair to a few Fair to most Fair to all 
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n. Do the instructional materials include adequate and appropriate uses of a variety of educa- 
tional technologies (e.g., video, computers, telecommunications)? 



1 



2 



3 



4 



5 



Little or no 

educational technology 
included 



Some appropriate 
educational technology 
included 



Many rich and useful 
applications of educational 
technology included 



o. What is the overall quality of the pedagogical design of these instructional materials? 

1 2 3 4 5 

Low Medium High 



IV. Implementation and System Support 

a. Will the teachers find the materials interesting and engaging? 



1 


2 


3 


4 5 


Dry and boring 




Somewhat interesting 
and engaging 


Interesting and engaging 
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b. Do the instructional materials include information and guidance to assist the teacher in 
implementing the lessons? 



1 


2 


3 


4 


5 


No teacher support 




Some teacher support 




Rich and useful 
teacher support 



c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the required mathematics materi- 



als? 










1 


2 


3 


4 


5 


No materials support 




Some materials support 




Rich and useful 
materials support 



d. Do the instructional materials provide information about the kinds of professional develop- 
ment experience needed by teachers to implement the materials? 

1 2 3 4 5 

Little or no Partial information Rich and useful 

information provided provided information provided 
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e. Do the materials provide guidance in how to link the materials with the district and state 
assessment frameworks and programs? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



f. Do the materials provide guidance and assistance for actively involving administrators, 
parents, and the community-at-large in supporting school mathematics? 



1 


2 


3 


4 5 


No guidance 




Some guidance 


Rich and useful guidance 



g. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 

1 2 3 4 5 

Teacher-unfriendly Somewhat teacher-friendly Teacher-friendly 
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V. Major Strengths and Weaknesses 

a. In your opinion, what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 



VI. Overall Quality, Value, and Contribution 

High 
4 5 

4 5 

4 5 

4 5 

4 5 

4 5 



a. 



In your opinion, what is the overall quality of these materials relative to: 

Low 

turning students on to mathematics? 12 3 

making students think? 12 3 

quality of mathematics content? 12 3 

quality of pedagogy? 12 3 

quality of classroom assessments? 12 3 

encouraging teachers to teach differendy? 12 3 
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b. In your opinion, what is the overall quality of these instructional materials? 

12 3 45 

Low Medium High 



c. To what extent would you encourage the dissemination, adoption, and implementation of 
this curriculum? 

1 2 3 4 5 

Not worthy of OK to disseminate, OK to disseminate, 

dissemination, adoption, adopt, and implement adopt, and implement 

nor implementation if revised as is 
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APPENDIX B 

This appendix contains a brief description of each set of instructional 
materials from the Middle School Science Study that are discussed in this 
report. The materials are grouped as follows: 

■ Comprehensive, multiple continuous years, grade 6 and beyond; 

■ Comprehensive, multiple continuous years, grades K-6; 

■ Comprehensive, integrated mathematics, science, and technology, 
grades 7-8; 

■ Single-year comprehensive; and 

■ Technology-driven supplemental, but material for at least a full year. 
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Comprehensive 6-10 

Title: PRIME Science 

Contact: Richard Saykally, University of California, Berkeley 

Publisher: Kendall/Hunt Publishing Co. 

4050 Westmark Drive 
P.O. Box 1840 
Dubuque, IA 52004 
1 (800) 258-5622 

PRIME Science provides a U.S. adaptadon of “Salter’s Science” — a well- 
tested British multidisciplinary science program for middle grades. The 
science is balanced — not integrated — between life, earth, and physical sci- 
ence, developing conceptual understanding and integrating mathematics, 
technology, and decision making. The science is rigorous, interesting, and 
useful to the student. Among the major integrative themes that provide 
structure for grades six through 10 are the earth in space, properties of 
matter, and so forth. Each unit begins with an applicadon. The teachers’ 
guides are directed at first-year teachers, not teaching in their major disci- 
pline. Included are student preconcepdons, safety, background, and ways of 
introducing the content and assessment items. The visually stimuladng, 
attractively designed student supplements for each of the 40 units contain 
the applicadon, a summary of what students should know, what they need to 
learn, and the acdvides they can do. The materials were tested and rewritten 
by teachers and science educators at several sites throughout the United 
States. Professors at the University of California, Berkeley, reviewed the 
materials for content accuracy. The Bridsh developers were part of the 
design team. Not only does the adaptation involve language translation, but 
also changing data to interest U.S. audiences and adding units to meet local 
interests and frameworks. 

Materials Reviewed: Sixth-grade materials developmental form were 
reviewed. 
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Strengths: 

1 . The program is planned to introduce science content at various 
grade levels and to revisit it as students advance to later grade levels. 
Content introduced in the sixth grade might be revisited several 
times in later grades, giving students opportunities to further de- 
velop their own understanding as they mature. 

2. The activity-based approach gives students experiences with numer- 
ous science concepts in a way they will more likely remember and 
understand. 

3. The teacher materials provide support for both the experienced 
teachers and those unfamiliar with the content being taught. It is 
flexible enough to allow teachers to supplement the curriculum with 
their own experiences and to integrate current events. 

General Concerns: 

1. Efforts should include more of the emerging technologies (e.g., CD- 
ROM). 

2. More of the student challenges should be open ended to develop 
inquiry skills. 

3. The program should include a greater variety of assessment items. 
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Comprehensive 6-8 



Title: 



Science 2000 

A Middle School Technology Based 
Curriculum Management Tool 
Ellen M. Nelson 
Decision Development Corp. 

2680 Bishop Drive 
Suite 122 

San Ramon, CA 94583 
(510) 830-8894 



Contact: 

Publisher: 



Science 2000 is a multimedia science curriculum for grades 5 to 8 that 
uses a constructive pedagogy, a thematic approach, and a multidisciplinary 
organization of science. At the sixth- to eighth-grade levels, four different 
themadc units integrate materials from earth, life, and physical science 
leading to a decision-making situation in which students apply knowledge to 
solve an STS-type problem. Grade six units are: Earth’s Changing Environ- 
ments, Growth and Development, Physics of Building, and Chemistry of 
Food. Eighth-grade units are: Genetics and Heredity, Sun and Global Cli- 
mate Change, Ears to the Sky, and Natural Disasters. The large ideas of 
science and science as a method of knowing are stressed. Each unit includes 
teacher lessons, student activities, and bibliographic resources stored in 
software; a learning resource management tool compatible with either IBM 
or Macintosh computers, which allows teachers to choose, write, and edit 
lessons, as well as assign student activities, access videodiscs, and review 
supplemental material; four videodiscs; eight hands-on activities; and models 
for pre- and in-service teacher education. Science 2000 correlates to the 
standards and has been adopted as a textbook alternative in Alabama, 

Florida, Georgia, Kentucky, Louisiana, Texas, Utah, and West Virginia. 

Materials Reviewed: Units in grades 6 and 8 were reviewed. 
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Strengths: 

1 . The professional quality of the programs and the software are high, 
video clips and databases are current, and themes and story lines are 
engaging and appropriate. 

2. The materials are diverse enough to support a variety of uses by 
both inquisitive students and creative teachers. 

3. There is very good alignment with standards, and the focus on four 
major questions per year allows in-depth study. The thematic, 
problem-solving approach is question driven, and there are some 
open-ended questions. 

General Concerns: 

1. The actual assessment pieces are vague. 

2. The program’s complexity (extensions and flexibility) make the 
program difficult to use. The technology was very difficult to set up. 
Teachers will need considerable in-service training to use this pro- 
gram effectively. 

3. The lack of text may cause problems with parents, and there are no 
materials for communicating with them. 
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Comprehensive 6-8 



Title: 

Contact: 

Publisher: 



Middle School Science and Technology 

Michael Doherty, Biological Sciences Curriculum Study 

Kendall/Hunt Publishing Co. 

4050 Westmark Drive 
P.O. Box 1840 
Dubuque, IA 52004 
1 (800) 258-5622 



Middle School Science and Technology is a three-year, activity-based, middle 
school program for grades 6 to 8 as a continuation of the Biological Sciences 
Curriculum Study (BSCS) K-6 program. It focuses on the development of the 
early adolescent, illustrates careers in science, and emphasizes reasoning and 
cridcal thinking. The content is structured around major themes of patterns 
of change, diversity and limits, and systems and change, with emphasis on 
personal dimensions of science and technology, science and technology in 
society, technological problem solving, and the nature of scientific explana- 
don. The content is strongest in the life sciences, but generally aligns well 



with the content recommendations in the National Science Education Standards 
(NSES). 



Materials Reviewed: Sixth-, seventh-, and eighth-grade materials were 

reviewed. 

Strengths: 

1. Emphasis is on development and use of good pedagogy; for ex- 
ample, the philosophy and approach to cooperadve learning is 
thoroughly explained and consistendy used. 

2. There is excellent attendon to professional development, teaching, 
program, system, assessment, as well as content; there is a high level 
of consistency with the standards. 

3. The “less is more” approach is used to build an accurate understand- 
ing of overriding concepts and related subconcepts. 
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General Concerns: 

1. The eighth-grade materials are not consistent with the format and 
approach so effectively used for grades six and seven* The quality of 
the introductory units is not equivalent to the other units. 

2. The teacher support materials tend to be wordy to the detriment of 
easy understanding. 

3. There is limited opportunity for students to develop and pursue 
questions of their own. 
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Comprehensive K-6 



Title: 

Contact: 

Publisher: 



Science for Life and Living 

Catherine Monson, Biological Sciences Curriculum Study 
Kendall/Hunt Publishing Co. 

4050 Westmark Drive 
P.O. Box 1840 
Dubuque, IA 52004 
1 (800) 258-5622 



Science for Life and Living is a K-6 elementary science program that 
encourages students to make decisions and take actions that will improve the 
quality of their lives. At each grade level, one major concept and one major 
skill integrate the disciplines, so that students can make meaningful connec- 
tions between areas of study. For example, Level 1 ideas and concepts 
including the following: Introduction to Order and Organization — Objects 
and Properties (science), Materials and Structures (technology), Safety and 
Security (health). Major ideas in Levels 2-6 include the following: Level 2 — 
Introduction to Change and Measurement; Level 3 — Introduction to Patterns 
and Predictions; Level 4 — Introduction to Systems and Analysis; Level 5 — 
Introduction to Energy and Investigation; Level 6 — Introduction to Balance 
and Decisions. The curriculum uses the Biological Sciences Curriculum 
Study instructional model, is based on constructivist learning theory, inte- 
grates cooperative learning, and includes kits of hands-on materials and an 
implementadon guide for use by personnel in school districts and school 
buildings. The components of the program consist of: (1) two teachers’ 
guides in three-ring binders for each level, K-6; (2) student materials for each 
level; (3) kits of hands-on materials for each level; and (4) an implementadon 
guide for administrators and leadership teams within schools. 

Materials Reviewed: Two levels, grades five (Energy and Invesdgation) 
and six (Balance and Decisions), of the curriculum were reviewed. Each level 
consisted of a teacher guide in the form of a notebook plus a student text. 

The Implementation Guide was also reviewed. 




^JIDEBOOK TO EXAMINE SCHOOL CURRICULA 



1051 



108 



ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE (J.S. CURRICULA 



Strengths: 

1 . Teacher guide takes the teacher step-by-step through the curricu- 
lum — valuable for the science-timid teacher. 

2. The materials udlize inexpensive supplies. 

3. Team skills are taught. 

4. The Implementation Guide for use by districts, principals, and schools 
is excellent. 

General Concerns: 

1. At times the content appears to be very thin. 

2. There is too much focus on terms, and the content diagrams are 
somewhat unfriendly. 

3. Hands-on, active learning is not always present; activities are more 
like demonstrations; mostly pencil-and-paper activities are used. 
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Comprehensive K-6 



Title: 

Contact: 

Publisher: 



Full Option Science System (FOSS) 

Lawrence F. Lowry, Lawrence Hall of Science 



EBEC 



310 South Michigan Avenue 
Chicago, IL 60604 
1 (800) 554-9862 



The Full Option Science System (FOSS) for grades K-6 is offered to schools 
as a collection of standalone modules on different topics appropriate for 
students in grades K-6. The module consists of a kit of student materials, a 
detailed teacher’s guide, and a teacher’s preparation video. The activities are 
organized into four strands: Life Science, Physical Science, Earth Science, 
and Scientific Reasoning and Technology. Five unifying themes weave 
through the modules of the program: Pattern, Structure, Interaction, 

Change, and System. Student assessment suggestions are included. Four 
modules in any academic year would easily constitute a complete curriculum. 
Eight modules (two in each strand) have been developed to be appropriate 
for students in sets of grades K-2, 3-4, and 5-6. There are two language 
versions of all student materials packaged in each guide, Spanish and En- 
glish. There are also suggestions for cultural enrichment, sensitivity to 
cultural difference, and sheltered instruction. FOSS employs cognitive and 
constructivist approaches to science instruction. Students work in collabora- 
tive groups of four to maximize effective use of materials and promote 
student-student interactions. Fundamental academic skills of language and 
mathematics are integrated into all activities, and guidance is provided to 
help teachers lead productive discussions. 

Materials Reviewed: Units reviewed were Life Science: Food and 
Nutrition, and Environments; Physical Science: Levers and Pulleys, and 
Mixtures and Solutions; Earth Science: Solar Energy, and Landforms. 
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Strengths: 

1. Materials are user friendly, clear, and clean; teachers’ guides and 
videos are excellent. 

2. Modules are built on a strong psychological and teaching research 
foundation. 

3. There is strong science content with a good balance of depth and 
breadth. 

4. Children will enjoy the cleverly designed activities and materials. 

5. Assessments are strong; questioning approaches direct teachers 
regarding the types of questions to ask. 

General Concerns: 

1. Many of the hands-on science activities are excellent, but materials 
do not encourage students to ask their own questions. 

2. Materials provide guided inquiry, but little open-ended inquiry. 

3. Bibliography and history of science are thin. 
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Comprehensive K-6 

Title: Insights: A Hands-On Inquiry Science Curriculum 

Contact: Karen Worth, Education Development Center 

Publisher: Optical Data Corporation 

30 Technology Drive 
Warren, NJ 07059 
1 (800) 248-8478 

The Education Development Center’s Insights is a hands-on, inquiry- 
oriented science program designed for self-contained elementary classrooms. 
The science is appropriate and current, and the supplies required are inex- 
pensive and easy to obtain. These materials are designed to improve stu- 
dents’ abilities to think critically, use language, and solve problems using the 
natural world as an experimental base. Since urban systems face extremely 
complex problems, the science materials are specifically aimed at these 
systems. There is a balance of life, physical, and earth sciences, tying the 
experimental base to the urban setting where appropriate. The materials 
integrate science with the rest of the elementary curriculum, pardcularly 
language arts and mathemadcs. 

Materials Reviewed: Six units were reviewed: Changes of State, grades 
4-5; The Mysterious Powder, grades 4-5; Reading the Environment, grades 
4-5; Structures, grade 6; There Is No Away, grade 6; and Human Body 
Systems, grade 6. 

Strengths: 

1. The topics are important for the age levels, with appropriate and 
current science. 

2. The supplies called for are inexpensive. 

3. Acdvides stretch over a period of time and allow exploration on the 
part of students. 

4. Effecdve sequencing of the curriculum within the units (i.e., acdvides) 
provides good guidance for teachers; assessments are also good. 
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General Concerns: 

1. Minor errors and inconsistencies exist and should be edited out 

2. The amount of information provided to teachers in the background 
text appears insufficient. 

3. Assessment questions need to focus on topics that are familiar/ 
accessible to all students (an example of a bicycle activity would leave 
those without bicycles at a disadvantage). 
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Comprehensive K-6 

Title: Science and Technology for Children 

Contact: Douglas M. Lapp, National Science Resource Center, 

Smithsonian Institution, and National Academy of Sciences 
Publisher: Carolina Biological Supply Co. 

2700 York Road 
Burlington, NC 27215 
1 (800) 227-1150 



The National Science Resource Center (NSRC) joined with the National 
Academy of Sciences and the Smithsonian Insdtudon to bring together 
teachers, educators, and scientists with a great diversity and richness of 
experience to create and disseminate an innovative elementary science 
program for grades one through six called Science and Technology for Children 
( STC '). Twenty-four hands-on, inquiry-centered units constitute a complete 
elementary science program for grades one through six. In addidon, there 
are 16 science readers to complement the 16 STC units for grades three 
through six. Each STC unit provides children with the opportunity for in- 
depth learning about topics in the physical, life, or earth sciences and tech- 
nology through direct observation and experimentation. The units invite 
children first to develop hypotheses and then to test their ideas just as profes- 
sional scientists do. Along the way, children develop patience, persistence, 
and confidence in their own ability to tackle and solve real problems. The 
teachers act as guides to the hands-on learning, encouraging students to 
explore new ideas for themselves and expand their understanding of the 
world around them. School districts are able to use these materials either 
collectively, as a complete elementary science program, or individually, as a 
supplement to an existing science program. These materials are designed to 
meet the needs of elementary school children from diverse cultural and 
ethnic backgrounds. 



Materials Reviewed: Of the 24 units, those considered appropriate for 
fifth and sixth grades were Food Chemistry, Electric Circuits, Ecosystems, 
Animal Studies, Microworlds, Experiments with Plants, Measuring Time, and 
Floating and Sinking! 
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Strengths: 

1. The program presents carefully sequenced, hands-on activities 
designed to lead to conceptual development. 

2. Rich, strong, and accurate science content is a real strength. 

3. The program is positively aligned with standards, including technol- 
ogy content and assessment. 

General Concerns: 

1. The materials are very teacher directed, with insufficient opportu- 
nity for students to ask and answer their own questions. 

2. The teachers’ guide has too much information that is hard to locate 
and use. 
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Integrated Comprehensive 7-8 



Title: 

Contact: 

Publisher: 



Integrated Mathematics , Science and Technology 

Francie Loepp, Illinois State University 

Glen Co. Macmillan 

3008 West Willow Knolls Drive 

Peoria, IL 61614 

(309) 438-3089 



The Integrated Mathematics, Science and Technology (IMaST) program is 
centered around the topics of biotechnology, manufacturing, and forecasting. 
Each unit includes objectives, experiential learning, appropriate use of 
multimedia, and appropriate technology and evaluation instruments. The 
materials motivate students, especially those from groups underrepresented 
in technological careers, to learn the foundation concepts of mathematics, 
science, and technology by involving them in enriched learning experiences 
relevant to their daily lives. The materials are designed to be used by math- 
ematics, science, and technology teachers concurrently over a nine-week 
school session. Assessment activities designed for mathematics, science, and 
technology are included. Though some content areas are not addressed, 
there is generally excellent content alignment with the National Science Educa- 
tion Standards (NSES). 

Materials Reviewed: Seventh-grade materials were reviewed. The 
eighth-grade materials are under development. 

Strengths: 

1. The materials and activities apply a hands-on approach. 

2. The content and activities in science, mathematics, and technology 
are integrated and there is a well-represented progression of ideas 
and skills. The technology and science content is current. 

3. The program provides teacher materials and activities as a basis for 
an integrative approach to learning. 
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General Concerns: 

1. The program’s activities and approaches are somewhat prescriptive. 

2. It is not clear that there is sufficient attention to activities for students 
with high potential in science, mathematics, and/or technology. 

3. The format of the program is not teacher friendly. 

4. Implementation of the program may be difficult. The design of the 
program — three teachers teaching the program concurrently — may 
not fit into the schedule of some schools. 
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Single- Year Comprehensive 



Title: 

Contact: 



Event-Based Science : Earth Science 
Russell Wright 

Montgomery County Public Schools 
850 Hungerford Drive 
Rockville, MD 20850 
Addison-Wesley Publishing Co. 



Publisher: 



Route 128 

Reading, MA 01867 
1 (800) 552-2259 



The materials in Event-Based Science: Earth Science (EBS) provide a year- 
long, event-based science curriculum for heterogeneously grouped middle 
school students in grades 6-9 for use primarily in departmentalized earth 
science classes. EBS is different from other approaches to science instruction 
and curriculum writing. The event focus (e.g., earthquakes, volcanoes, 
tornadoes) makes each unit topical and relevant to early adolescents. It 
allows science to become less compartmentalized. It allows for a natural 
highlighting of non-traditional roles filled by women and minorities. High- 
interest activities are models for other activities. The approach taken by EBS 
requires students to explore other sources of information (biographies, 
newspapers) in order to complete class assignments. EBS stresses alternative 
assessment techniques and grading strategies that reward success and 
downplay failure. Nationally disseminated products include a textbook, a 
teacher’s resource notebook, and videotape and/or videodisc support. 

Materials Reviewed: All current modules and two pilot test modules 
were reviewed. 

Strengths: 

1. The materials have a strong inquiry focus. 

2. The materials are highly student centered with relevant tasks. 

3. There are good uses of authentic assessment. 

4. The modules are interchangeable. 
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General Concerns: 

1 . There are a limited number of science activities. 

2. The content of the videos could be more content-rich, but the 
current “hook” they provide is well done. 

3. Some pilot test modules contain content errors and are not gener- 
ally as engaging as the earlier work. 
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Single- Year 



Comprehensive 



Title: 

Contact: 

Publisher: 



Science Education for Public Understanding Program (SEPUP): 
Issues-Oriented Science for Secondary Schools 
Herbert D. Their, University of California, Berkeley 
LAB-AIDS, Inc. 

17 Colt Court 
Ronkonkoma, NY 11779 
(516) 737-1133 



The Science Education for Public Understanding Program (SEPUP) materials 
support two one- year courses of study: a concrete course for middle school 
and a course emphasizing global issues for high school. The courses stress 
issue-oriented science and the use of scientific evidence and risk-benefit analy- 
sis in making decisions. These courses continue the emphasis of the Chemical 
Education for Public Understanding Program ( CEPUP % societal issues involving 
the use of chemicals, and expand the scope of dealing with other issues of life, 
earth and physical sciences, and technology. Eight modules cover many of the 
large themes of science proposed in The American Association for the Ad- 
vancement of Science’s Project 2061 along with issue-oriented themes such as 
evidence-based decision making, uncertainty and controversy, and science and 
societal systems. Materials include a teacher’s resource book, a student text, 
projects and extension activities, kits, videotapes, and software. Assessment of 
student learning is built into these materials. Note: A set of life science mod- 
ules for SEPUP is now under development. 

Materials Reviewed: All four sections of Issues , Evidence , and You were 
reviewed. 



Strengths: 

1. The materials are engaging, provide good activities for student decision 
making, and offer opportunities for student-designed inquiry. 

2. The scope and sequence allow for conceptual growth. 

3. There is an excellent assessment component. 
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General Concerns: 

1. Materials cannot be used as a “full” curriculum; additions are 
needed in the areas of life and earth science. 

2. There is limited use of educational technology. 
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Single- Year Comprehensive 

Title: Chemical Education for Public Understanding Project 

Contact: Herbert D. Their, University of California, Berkeley 

Publisher: LAB-AIDS, Inc. 

17 Colt Court 
Ronkonkoma, NY 11779 
(516) 737-1133 

The Chemical Education for Public Understanding Project (CEPUP) at the 
Lawrence Hall of Science has developed 12 modular sets of interdisciplinary 
materials for use at the middle/junior high school level that can comprise a one- 
year course. The content is up-to-date, is accurate, and gives students opportuni- 
ties to study materials in depth through active application of concepts. The 
materials introduce students to scientific concepts in chemistry and their interac- 
tion with people and the environment. CEPUP materials highlight areas of 
direct societal concerns associated with science and technology. Students are 
given chemistry-based laboratory investigations and experiments that focus on 
the environment, biotechnologies, industrial processes, agricultural practices, 
alternative energy sources, and health science. 

Materials Reviewed: The Teacher's Guide , including student sheets, and 
Guide for Implementation were reviewed. All 12 of the modules were examined. 

Strengths: 

1. The materials address real-life issues and give students multiple 
opportunities to apply chemistry. 

2. The focus on interdisciplinary topics is one that many middle school 
teachers and students will find appealing. 

3. The modular format is a strength. 

4. These materials help students to develop good data skills. 
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General Concerns: 

1. The classroom assessments emphasize written tests; many potential 
alternative assessments are not developed. 

4. The materials emphasize scientific processes and do not sufficiently 
emphasize scientific theory and models. 
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Single-Year Comprehensive 



Title: 

Contact: 

Publisher: 



Junior High/ Middle School Life Science Program 
Harold Pratt, Jefferson County Public Schools 
Kendall/Hunt Publishing Co. 

4050 Westmark Drive 
P.O. Box 1840 
Dubuque, IA 52004 
1 (800) 258-5622 



The Jefferson County, Colorado Public School System developed mate- 
rials that constitute a year-long junior high/middle school program in life 
science that emphasizes the understanding and care of the human body. The 
development was done in close cooperation with the University of Colorado 
Health Sciences Center and with the support of local physicians and univer- 
sity-level scientists and science educators. The program provides an alterna- 
tive for teachers and schools seeking materials to improve their life science 
curriculum and serves as a resource for schools seeking to integrate health 
topics with their existing life science course. General topics included in the 
materials are life science, human biology and reproduction, ecology, cells, 
and genetics. The life science section, although only in the context of human 
biology, aligns well with the National Science Education Standards (NSES). Part 
of the program directs students toward an ability to make decisions in and 
about their local environment. Materials include student text and investiga- 
tions, an extensive Teachers’ Guide, the Teachers’ Resource Book (which includes 
transparency masters, worksheets, etc.), and the Guide for Implementation. 

Materials Reviewed: The student text and investigadons. Teacher’s 
Guide, Teacher’s Resource Book, and the Guide for Implementation were reviewed. 
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Strengths: 

1. The materials contain a strong activity orientation, and many stu- 
dents will find them engaging. 

2. The materials explicitly help students move from the big picture to 
smaller ideas. 

3. The materials provide good teacher support. 

4. The health-related topics contain sensitive treatment of key issues in 
language appropriate for middle school students. 

General Concerns: 

1. The materials are weak in the area of student assessments. 

2. Students are not encouraged to design their own investigations. 

3. This curriculum is overstuffed — there is more material presented 
than students can reasonably learn, and much of it focuses on facts 
and vocabulary. 
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Supplemental Technology Driven 

Title: National Geographic Kids Netivork 

Contact: Daniel Barstow, TERC, Inc. 

Publisher: National Geographic Society 

Educational Services 
Department 89 
Washington, DC 20036 
1 (800) 368-2728 

These materials extend those developed for grades 4-6 with nine units 
(or approximately 90 weeks) of supplementary science material targeted for 
grades 7-9 and organized around telecommunications-based collaborative 
student research. The materials contain coordinated curriculum and soft- 
ware and were designed by TERC in collaboration with the National Geo- 
graphic Society (NGS). Each unit requires students to gather data, share 
these data with students in other school districts over a telecommunications 
network, and analyze the collected data. This approach allows students to 
perform like scientists. Before gathering the data, students study the under- 
lying science content and learn the experimental skills required to perform 
appropriate measurements. Following data collection, students apply data 
analysis techniques and reflect on the social significance of the problem 
addressed in the study. Study areas proposed include Conditions for Growth, 
Trees, Student Fitness, Acid Deposition, Recycling and Composting, Radon, 
Alternative Energy Sources, Automobile Accidents, and Greenhouse Gases. 
The materials include a teacher’s guide, readings, student lab sheets, assess- 
ments, overhead transparencies, posters, works and reference disks, and a 
Quick Guide to Using NGS Works. 

Materials Reviewed: The unit “What Is Our Soil Good For?” was reviewed. 
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Strengths: 

1. Technology is used well as a tool in these materials. 

2. There is good focus on science as inquiry. 

3. The data collection/analysis activities are strong. 

4. The materials permit students to explore science experiences in depth. 

General Concerns: 

1. The materials are expensive, the software is complicated, and diffi- 
culties can be anticipated in its use. 

2. The technology is not used to its full potential. There is a lack of 
graphics, the menus are tedious, and data analysis tools are weak. 

3. These materials do not compose a full-year program. 





GUIDEBOOK TO EXAMINE SCHOOL CURRICULA 



ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE US. CURRICULA 



The Project 2061 
Curriculum-Analysis Procedure 



Introduction 

Deciding which curriculum materials to use is one of the most important 
professional judgments that educators make. Textbook adopdon committees 
make recommendations that influence instruction for years to come, and the 
daily decisions teachers make about which teaching units or chapters to use — 
and how to use them — largely determine what and how students will be 
expected to learn. 

Such important decisions require a valid and reliable method for evalu- 
ating the quality of curriculum materials. Even an in-depth review of the 
topics covered by a textbook or a teaching unit may not be sufficient to 
determine whether the material will actually help students learn that content. 
What is needed is a manageable process for examining curriculum materials 
that gets below the surface by focusing intensely on the appropriateness of 
content and the utility of instructional design. 

With funding from the National Science Foundation and in collabora- 
tion with hundreds of K-12 teachers, curriculum specialists, teacher educa- 
tors, sciendsts, and materials developers, Project 2061 of the American 
Association for the Advancement of Science (AAAS) has been developing a 
process for analyzing curriculum materials. Field tests suggest that Project 
2061 ’s curriculum-analysis procedure will not only serve the materials adop- 
tion needs of the schools but also help teachers revise existing materials to 
increase their effectiveness, guide developers in the creadon of new materi- 
als, and contribute to the professional development of those who use it. 

Specific Learning Goals Are Key 

Until recently, there was nothing against which to judge appropriateness 
of content and udlity of instructional design. Now, as a result of the stan- 
dards-based reform movement in education, these judgments can be made 
with a high degree of confidence. In science and mathematics, for example, 
the appearance of Science for All Americans (AAAS, 1989), Curriculum and 
Evaluation Standards for School Mathematics (National Council of Teachers of 
Mathematics, 1989), Benchmarks for Science Literacy (AAAS, 1993), and National 
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Science Education Standards (National Research Council, 1996) has made it 
possible to make more thoughtful decisions about curriculum materials than 
ever before. 

Although the Project 2061 curriculum-analysis procedure was developed 
using the learning goals in Benchmarks and the mathematics and science 
standards, subsequent work has indicated that some state education frame- 
works also can be used. Indeed, the process would seem to apply to any K-12 
school subject for which specific learning goals have been agreed upon. 

These goals must be explicit statements of what knowledge and skills stu- 
dents are expected to learn, and they must be precise. Vague statements such 
as “students should understand energy” are not adequate. Instead, consider 
this benchmark dealing with energy-related concepts that students should 
know by the end of the eighth grade: 

Most of what goes on in the universe — from exploding stars and biological 
growth to the operation of machines and the motion of people — involves some 
form of energy being transformed into another. Energy in the form of heat is 
almost always one of the products of an energy transformation. 

Similar explicit statements can be found in the fundamental concepts of 
the National Research Council’s National Science Education Standards (NSES). 

At its simplest level, the Project 2061 curriculum-analysis procedure 
involves the following five steps: 

■ Identify specific learning goals to serve as the intellectual basis 
for the analysis. This is done before beginning to examine any 
curriculum materials. The source for appropriate goals can be 
national standards or benchmark documents such as those men- 
tioned above, state or local standards and curriculum frameworks, or 
sources like them. To be useful, the goals must be precise in describ- 
ing the knowledge or skills they intend students to have. If the set of 
goals is large, a representative sample of them should be selected for 
purposes of analysis. 

■ Make a preliminary inspection of the curriculum materials to see 
whether they are likely to address the targeted learning goals. If 

there appears to be little or no correspondence, the materials can be 
rejected without further analysis. If the outlook is more positive, go 
on to a content analysis. 
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■ Analyze the curriculum materials for alignment between content 
and the selected learning goals. The purpose here is to determine, 
citing evidence from the materials, whether the content in the 
material matches specific learning goals — not just whether the topic 
headings are similar. At the topic level, alignment is never difficult, 
since most topics — heredity, weather, magnetism, and so forth — lack 
specificity, making them easy to match. If the results of this analysis 
are positive, then reviewers can take the next step. 

■ Analyze the curriculum materials for alignment between instruc- 
tion and the selected learning goals. This involves estimating the 
degree to which the materials (including their accompanying 
teacher’s guides) reflect what is known generally about student 
learning and effective teaching and, more important, the degree to 
which they support student learning of the specific knotvledge and skills 
for which a content match has been found. Again, evidence from the 
materials must be shown. 

■ Summarize the relationship between the curriculum materials 
being evaluated and the selected learning goals. The summary can 
take the form of a profile of the selected goals in terms of the con- 
tent and instruction criteria, or a profile of the criteria in terms of 
the selected goals. In either case, a statement of strengths and 
weaknesses should be included. With this information in hand, 
reviewers can make more knowledgeable adoption decisions and 
suggest ways for improving the examined materials. 

In addidon to its careful focus on matching content and instruction to 
very specific learning goals, the Project 2061 procedure has other features 
that set it apart. For example, its emphasis on collecdng explicit evidence 
(cidng page numbers and other references) of a material’s alignment with 
learning goals adds rigor and reliability to decisions about curriculum materi- 
als. Similarly, the Project 2061 procedure calls for a team approach to the 
analytical task, thus providing opportunities for reviewers to defend their 
own judgments about materials and to question those of other reviewers. 

These and other characteristics help make participation in the analytical 
process itself a powerful professional development experience. 
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The Project 2061 Curriculum-Analysis Procedure in Detail 

To provide a better sense of how the procedure works, the following 
describes in more detail each step in the procedure, using learning goals 
from Project 2061 ’s Benchmarks for Science Literacy as an illustrative frame of 
reference. The description pays particular attention to the various criteria 
used to evaluate the instructional effectiveness of materials. 

Identify specific learning goals to serve as the intellectual basis for 
the analysis. After reviewers have agreed upon a set of learning goals as a 
framework for the analysis (in this case, the benchmarks in Benchmarks for 
Science Literacy) , the task is then to choose specific benchmarks that will serve 
as the focus of further study. 

When evaluating standalone curriculum units that cover a relatively short 
period of time, it might be possible and worthwhile to analyze all of the bench- 
marks that appear to be targeted by the material. However, in the evaluation of 
year-long courses or multiyear programs, this becomes impractical. Therefore, 
a crucial step in the analysis procedure is the sampling of a few benchmarks 
that will lead to valid and reliable generalizations about the material. 

Sampling of benchmarks should be representative of the whole set of 
goals specified in Benchmarks for Science Literacy and should reflect the review- 
ers’ needs. For example, if the review committee’s task is to select a course in 
high school Biology that is aligned with Benchmarks , it might identify a 
sample of benchmarks from life science sections in Benchmarks (e.g., cells, 
heredity, and evolution) and from other sections (e.g., nature of scientific 
inquiry, models, and communication skills). When examining middle-school 
science materials, one would probably want to broaden the range of bench- 
marks examined to include some from physical and earth science topics (e.g., 
energy, forces, and processes that shape the earth). 

Make a preliminary inspection of the curriculum materials to see 
whether they are likely to address the targeted learning goals. Once bench- 
marks have been selected, the next step is to make a first pass at the materials to 
identify those whose content appears to correspond reasonably well to Bench- 
marks. Materials that do not meet these initial criteria are not analyzed further. 

Reviewers then examine materials on the shortened list more carefully to 
locate and record places where each selected benchmark seems to be targeted 
(e.g., particular readings, experiments, discussion questions). If several 
sightings are found for some or all of the sample benchmarks in the material, 
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then these sightings will be looked at more carefully in subsequent steps of the 
analysis. If, on the other hand, sightings cannot be found for a significant 
number of the sample benchmarks, then the material is dropped from the list. 

Analyze the curriculum materials for alignment between content and 
the selected learning goals. This analysis is a more rigorous examination of 
the link between the subject material and the selected learning goals and 
involves giving precise attention to both ends of the match — the precise 
meaning of the benchmark on one end and the precise intention of the 
material on the other. 

With respect to each of the sampled benchmarks, the material is exam- 
ined using such questions as: 

■ Does the content called for in the material address the substance of 
a specific benchmark or only the benchmark's general “topic”? 

■ Does the content reflect the level of sophistication of the specific 
benchmark, or are the activities more appropriate for targeting 
benchmarks at an earlier or later grade level? 

■ Does the content address all parts of a specific benchmark or only 
some? (While it is not necessary that any particular unit would 
address all of the ideas in a benchmark or standard, the K-12 cur- 
riculum as a whole should do so. The purpose of this question is to 
provide an account of precisely what ideas are treated.) 



In addition, an attempt is made to estimate the degree of overlap be- 
tween the material's content and the set of benchmarks of interest. Thus, this 
step in the analysis is designed to answer questions regarding the material's 
inclusion of content that is not required for reaching science literacy and the 
extent to which the material distinguishes between essential and non-essen- 
tial content. (While distinguishing content essential for literacy from non- 
essential content in material might seem to be a luxury, it assists teachers in 
determining the range of students for which the material can be used. Identi- 
fying the non-essential material makes it easier for the teacher to direct 
better students to enrichment activities and allows students themselves to 
avoid overload from ideas that go beyond what is vital.) 

Analyze the curriculum materials for alignment between instruction 
and the selected learning goals. The purpose here is to estimate how well 
material addresses targeted benchmarks from the perspective of what is 
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known about student learning and effective teaching. The criteria for making 



the judgments in the instructional analysis are derived from research on 
learning and teaching and on the craft knowledge of experienced educators. 
In the context of science literacy, these are summarized in Chapter 13, 
“Effective Learning and Teaching,” of Science for All Americans ; in Chapter 15, 
“The Research Base,” of Benchmarks for Science Literacy, and in Chapter 3, 
“Science Teaching Standards,” of National Science Education Standards. 

From these sources, seven criteria clusters (shown below) have been 
identified to serve as a basis for the instructional analysis (for the specific 
questions within each cluster, see Appendix on page 137). The proposition 
here is that (1) the analysis would tie the instruction to each one of the 
sample benchmarks rather than look at instructional strategies globally and 
(2) in the ideal, all questions within each cluster would be well-addressed in 
any material — they are not alternatives. 

Cluster I. Providing a Sense of Purpose: Part of planning a coherent 
curriculum involves deciding on its purposes and on which learning 
experiences will likely contribute to those purposes. But while coher- 
ence from the curriculum designers’ point of view is important, it may 
not give students an adequate sense of what they are doing and why. 

This cluster includes criteria to determine whether the material 
attempts to make its purposes explicit and meaningful to students, 
either by itself or by instructions to the teacher. 

Cluster II. Taking Account of Student Ideas: Fostering better 
understanding in students requires taking time to attend to the 
ideas they already have, both ideas that are incorrect and ideas that 
can serve as a foundation for subsequent learning. Such attention 
requires that teachers be informed about prerequisite ideas/skills 
needed for understanding a benchmark and what their students’ 
initial ideas arejin particular, the ideas that may interfere with 
learning the scientific information. Moreover, teachers can help 
address students’ ideas if they know what is likely to work. This 
cluster examines whether the material contains specific suggestions 
for identifying and relating to student ideas. 
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Cluster III. Engaging Students with Phenomena: Much of the point 
of science is explaining phenomena in terms of a small number of 
principles or ideas. For students to appreciate this explanatory power, 
they need to have a sense of the range of phenomena that science can 
explain. “Students need to get acquainted with the things around 
them — including devices, organisms, materials, shapes, and num- 
bers — and to observe them, collect them, handle them, describe them, 
become puzzled by them, ask questions about them, argue about 
them, and then try to find answer's to their questions.” ( Science for All 
Americans , p. 201) Furthermore, students should see that the need to 
explain comes up in a variety of contexts. 

Cluster IV. Developing and Using Scientific Ideas: Science for All 
Americans includes in its definition of science literacy a number of 
important yet quite abstract ideas (e.g., atomic structure, natural 
selection, modifiability of science, interacting systems, common laws 
of motion for earth and heavens). Such ideas cannot be inferred 
directly from phenomena, and the ideas themselves were developed 
over many hundreds of years as a result of considerable discussion 
and debate about the cogency of theory and its relationship to 
collected evidence. Science literacy requires that students see the 
link between phenomena and ideas and see the ideas themselves as 
useful. This cluster includes criteria to determine whether the 
material attempts to provide links between phenomena and ideas 
and to demonstrate the usefulness of the ideas in varied contexts. 

Cluster V. Promoting Student Reflection: No matter how clearly 
materials may present ideas, students (like all people) will assign 
their own meanings to them. Constructing meaning well is aided by 
having students (1) make their ideas and reasoning explicit, (2) hold 
them up to scrutiny, and (3) recast them as needed. This cluster 
includes criteria for determining whether the material suggests how 
to help students express, think about, and reshape their ideas to 
make better sense of the world. 
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Cluster VI. Assessing Progress: There are several important rea- 
sons for monitoring student progress toward specific learning goals. 
Having a collection of alternatives can ease the creative burden on 
teachers and increase the time available to analyze student responses 
and make adjustments in instruction based on those responses. This 
cluster includes criteria for evaluating whether the material includes 
a variety of goal-relevant assessments. 



Cluster VII. Enhancing the Learning Environment: Many other 
important considerations are involved in the selection of curriculum 
materials — for example, the help they provide to teachers in encour- 
aging student curiosity and creating a classroom community where 
all can succeed, or the material’s scientific accuracy or attractiveness. 
The criteria listed in this cluster provide reviewers with the opportu- 
nity to comment on these and other important features. 



Summarize the relationship between the curriculum materials being 
evaluated and the selected learning goals. In the preliminary inspection, a few 
benchmarks were selected as representative of the set of goals that the material 
appears to target. Having analyzed whether the content in the material 
matches these specific benchmarks and how well the instructional strategies in 
the material support students learning these benchmarks, the final step in the 
process is to provide a profile of the material based on this analysis. 

The analysis makes it possible to produce two sets of profiles. The first 
illustrates how well the material treats each benchmark (for which a content 
match was found) across all criteria examined in the instructional analysis. 
Based on these profiles, conclusions can be made about what the material 
under consideration can be expected to accomplish in terms of benchmarks. 
For example, the profiles may indicate that the material treats one of the 
examined benchmarks well and the rest only moderately or poorly. 

The second set of profiles illustrates how well the material meets each 
criterion in the instructional analysis tool across all benchmarks examined. 

These profiles point to major strengths and weaknesses in the instructional 
design of the material. For example, the profiles may indicate that the material 
consistently includes appropriate experiences with phenomena relevant to the 
benchmarks but only occasionally provides students with opportunities to 
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reflect on these experiences. Depending on the time available and their inter- 
ests, a review committee could decide to produce either one or both sets of 
profiles. Profiles of different materials provide the basis for selecdon decisions. 

Support for Users 

Project 2061 is in the process of developing “Resources for Science 
Literacy: Curriculum Evaluation,” a CD-ROM that will offer full instruction 
in using the procedure. The CD-ROM and its print companion volume will 
contain (1) detailed instructions for evaluating curriculum materials in light 
of Benchmarks , national standards, or other learning goals of comparable 
specificity; (2) case-study reports illustrating the application of the analysis 
procedure to a variety of curriculum materials; (3) a utility for relating find- 
ings in the case-study reports to state and district learning goals; and (4) a 
discussion of issues and implications of using the procedure. 

Project 2061 also offers introductory workshops and longer training 
institutes to groups of educators. Typically three to six days long, the training 
institutes can be adapted to suit a variety of needs and time constraints. The 
project has offered customized workshops for K-12 science and mathematics 
teachers, teacher educators, school and university administrators, developers 
of curriculum materials, and policy makers. Depending on the interests of 
pardcipants, the workshops can focus on understanding learning goals, 
selecting materials, revising existing materials, or evaluating curriculum 
frameworks, among other possibilides. 

For information on Project 2061 ’s workshops and institutes (or any 
aspects of Project 2061’s work) contact Mary Koppal, Project 2061, American 
Associadon for the Advancement of Science (see the Guide to Using the 
Methods of Analysis section of this Guidebook for contact information). 
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Putting the Project 2061 Curriculum-Analysis Procedure to Work 

Many of the educators involved in developing and field testing the 
Project 2061 procedure have begun to use it to decide on materials for their 
classrooms, school districts, or states; to identify shortcomings in materials 
they are using; and to suggest ways to improve them. Here are a few ex- 
amples of how educators have adapted the procedure to their local needs and 
time constraints: 

San Antonio. Faced with the task of selecdng a new high-school 
physical science textbook from five possible choices, a San Antonio 
school district committee requested training in the Project 2061 
curriculum-analysis procedure. Already familiar with Benchmarks for 
Science Literacy and Science for All Americans, these 12 educators spent 
two days studying Project 2061’s analytical criteria, as well as some 
additional criteria decided locally. Committee members then evalu- 
ated one material apiece, with at least two committee members inde- 
pendently evaluating each material. When finished with their inde- 
pendent evaluations, those educators reviewing the same material met 
to compare their results and to come to an agreement about the value 
of the material. Then, about three weeks after the initial training, the 
whole group reconvened to share their results and settle on the 
material. Because the evaluation procedure requires reviewers to cite 
evidence for judgments made, the reviewers were prepared to justify 
their recommendations, pointing to specific instructional strategies 
for particular learning goals in physical science. 

After much discussion, the reviewers reached agreement on one 
material for the district. Throughout the process, the reviewers were 
very reflective and motivated by the work at hand. In fact, because 
the evaluation procedure had revealed some weaknesses even in the 
material they agreed to select, they decided to write a supplemental 
unit on one topic. 
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Philadelphia. The Philadelphia school district was already committed 
to teaching toward specific learning goals derived from Benchmarks 
and National Science Education Standards when it set out to identify 
materials that are aligned with those goals. Their list of possibilides 
included some materials developed through National Science Foun- 
dadon funding and some materials that had been favorably evaluated 
by the Project 2061 pilot and field test pardcipants. The district held 
training insdtutes to introduce teachers to the evaluadon procedure 
and to develop evaluadon skills that they would use to select materials 
from the list of possibilides. More than 200 teachers pardcipated in 
the insdtutes, giving the district a cadre of leaders who could assist in 
the school-based selecdon of curriculum materials. 

After employing the procedure to select materials for use in 
their classrooms, teachers planned to make a more thorough evalua- 
don of the materials when they eventually put them to use. Findings 
from the procedure also will be used to improve materials currently 
being implemented in district classrooms. Such remedies might 
include developing questions to focus students’ reflection on bench- 
mark ideas, adding activities to address student learning difficulties, 
or demonstrating how benchmark ideas are useful for making sense 
of the students’ world outside the classroom. 

Through its work with the Project 2061 procedure, Philadelphia 
has developed a group of educators who are becoming more knowl- 
edgeable about specific learning goals in Benchmarks and the Na- 
tional Science Education Standards and about the analysis criteria used 
to judge materials in light of these goals. As new, better aligned 
materials become available, the district will have a cadre of informed 
consumers who can recognize them and put them to work. Most 
important, district classrooms will reflect teaching and learning that 
engage all students in achieving science literacy goals essential for 
the 2lst century. 

Kentucky. In the fall of 1996, Project 2061 began to work with the 
director of the Kentucky Middle Grades Mathematics Teacher 
Network to adapt the project’s curriculum analysis procedure to 
mathematics. The Kentucky Network, which already reaches some 
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2,000 teachers, aims to align the state’s mathematics content and 
teaching practices in fifth through eighth grades with the recom- 
mendations of the National Council of Teachers of Mathematics’ 
Curriculum and Evaluation Standards for School Mathematics. In par- 
ticular, the network helps to train teachers in reviewing and analyz- 
ing curriculum materials so that they can (1) discriminate between 
materials that only claim to align with the mathematics standards 
and those that actually do and (2) recognize standards in the newer, 
integrated mathematics curricula. 

As the criterion for alignment, Project 2061 has used Kentucky’s 
Mathematics Care Content for Assessment (which elaborates the national 
mathematics standards into more specific goal statements) to analyze 
five middle-school curriculum projects funded by the National Sci- 
ence Foundation (NSF). While developing the analysis procedure and 
applying it to the materials, Project 2061 received continual feedback 
from Kentucky teachers and from a national advisory committee that 
included the developers of the NSF-funded curricula. 

During a 1997 two-week summer workshop, 32 Kentucky teach- 
ers used the analysis procedure and case-study reports to examine 
middle-grade mathematics materials and develop workshops for 
teachers throughout the state. In doing so they (1) gained a better 
understanding of integrated, problem-based mathematics curricula; 
(2) developed the skills necessary to evaluate mathematics curricula in 
light of specific learning goals; and (3) developed skills necessary to 
effectively share what they have learned throughout their regions. 

The workshop participants worked during the 1997-1998 school year 
with teachers, schools, and districts in their regions to assist in analyz- 
ing and evaluating mathematics curriculum materials. 
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APPENDIX 

Criteria for Instructional Analysis 

Project 2061 ’s curriculum-analysis procedure uses the following questions, 
grouped within seven criteria clusters, to determine the extent to which the 
material’s instrucdonal strategy is likely to help students learn the content. 

Cluster I. Providing a Sense of Purpose: 

Framing. Does the material begin with important focus problems, 
issues, or questions about phenomena that are interesting and/or 
familiar to students? 

Connected sequence. Does the material involve students in a 
connected sequence of activities (vs. a collection of activities) that 
build toward understanding of the benchmark(s)? 

Fit of frame and sequence. If there is both a frame and a connected 
sequence, does the sequence follow well from the frame? 

Activity purpose. Does the material prompt teachers to convey the 
purpose of an activity and its relationship to the benchmarks? Does 
each activity encourage each student to think about the purpose of the 
activity and its relationship to specific learning goals? 

Cluster II. Taking Account of Student Ideas: 

Identifying prerequisite knowledge/skills. Does the material 
specify prerequisite knowledge/skills that are necessary to learn the 
benchmark (s)? 

Alerting to commonly held ideas. Does the material alert teachers to 
commonly held student ideas (both troublesome and helpful) such as 
those described in Benchmarks, Chapter 15, “The Research Base”? 
Assisting the teacher in identifying students’ ideas. Does the 
material include suggestions for teachers to find out what their 
students think about familiar phenomena related to a benchmark 
before the scientific ideas are introduced? 

Addressing commonly held ideas. Does the material explicidy 
address commonly held student ideas? 
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Assisting the teacher in addressing identified students’ ideas. 

Does the material include suggestions for teachers on how to ad- 
dress ideas that their students hold? 

Cluster III. Engaging Students with Phenomena: 

Firsthand experiences. Does the material include activities that 
provide firsthand experiences with phenomena relevant to the 
benchmark when practical and, when not practical, make use of 
videos, pictures, models, simulations, and so forth? 

Variety of contexts. Does the material promote experiences in 
multiple contexts so as to support the formation of generalizations? 
Questions before answers. Does the material link problems or 
questions about phenomena to solutions or ideas? 

Cluster IV. Developing and Using Scientific Ideas: 

Building a case. Does the material suggest ways to help students 
draw from their experiences with phenomena, readings, activities, 
and so forth to develop an evidence-based argument for benchmark 
ideas ? (This could include reading material that develops a case.) 
Introducing terms. Does the material introduce technical terms only 
in conjunction with experience with the idea or process and only as 
needed to facilitate thinking and promote effective communication? 
Representing ideas. Does the material include appropriate repre- 
sentations of scientific ideas? 

Connecting ideas. Does the material explicitly draw attention to 
appropriate connections among benchmark ideas (e.g., to a concrete 
example or an instance of a principle or generalization, to an analo- 
gous idea, or to an idea that shows up in another field)? 
Demonstrating/modeling skills and use of knowledge. Does the 
material demonstrate/model or include suggestions for teachers on 
how to demonstrate/model skills or the use of knowledge? 
Encouraging practice. Does the material provide tasks/ questions 
for students to continue practicing skills or using knowledge in a 
variety of situations? 
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Cluster V. Promoting Student Reflection: 

Expressing ideas. Does the material routinely include suggestions 
(such as group work or journal writing) for having each student 
express, clarify, justify, and represent his or her ideas? Are sugges- 
tions made for when and how students will get feedback from peers 
and the teacher? 

Reflecting on activities. Does the material include tasks and/or 
question sequences to guide student interpretation and reasoning 
about phenomena and activities? 

Reflecting on when to use knowledge and skills. Does the material 
help or include suggestions on how to help students know when to 
use knowledge and skills in new situations? 

Self-monitoring. Does the material suggest ways to have students 
check their own progress and consider how their ideas have changed 
and why? 

Cluster VI. Assessing Progress: 

Alignment to goals. Assuming a content match of the curriculum 
material to this benchmark, are assessment items included that 
match the content? 

Application. Does the material include assessment tasks that re- 
quire application of ideas and avoid allowing students a trivial way 
out, like using a formula or repeating a memorized term without 
understanding? 

Embedded. Are some assessments embedded in the curriculum 
along the way, with advice to teachers as to how they might use the 
results to choose or modify activities? 

Cluster VII. Enhancing the Learning Environment: 

Teacher content learning. Would the material help teachers im- 
prove their understanding of science, mathematics, and technology, 
as well as their interconnections? 

Classroom environment. Does the material help teachers to create a 
classroom environment that welcomes student curiosity, rewards creativ- 
ity, encourages a spirit of healthy questioning, and avoids dogmatism? 
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Encouragement for all students. Does the material help teachers to 
create a classroom community that encourages high expectations for 
all students, that enables all students to experience success, and that 
provides different kinds of students with a feeling of belonging in 
the science classroom? 

Connections beyond the unit. Does the material explicitly draw 
attention to appropriate connecdons to ideas in other units? 

Other strengths. What, if any, other features of the material are 
worth noting? 
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California Department of Education 
Curriculum Frameworks and 
Instructional Resources 



K-8 Mathematics Instructional Resources Evaluation Form 
(Approved by the State Board of Education on April 1 1 , 1 996) 

This evaluation form will be used by the Instructional Resources Evalua- 
tion Panel (IREP) as they review K-8 resources. The criteria are based on 
Appendix A of the Mathematics Frameivork for California Public Schools , Kinder- 
garten Through Grade Twelve , approved by the State Board of Education on 
November 8, 1991, and are designed to be used with complete programs (at 
least a full grade level). This form is different from the evaluation form used 
for the 1994 primary adoption. It has been revised to include the impor- 
tance of a “...mathematics program that reflects a balance of basic skills, 
conceptual understanding, and problem solving.” The concept of the “bal- 
anced program” has been added to the following categories: (1) Mathemati- 
cal Content; (2) The Work Students Do; (3) Program Organization and 
Structure; and (4) Assessment. 

The criteria are organized into the following categories. The percent 
that each category contributes to the overall score is listed in parentheses. 

■ Mathematical Content: Which mathematical ideas and subject 
matter provide the basis for the instructional program. (20%) 

■ Program Organization and Structure: How the program is orga- 
nized into a year’s work of cohesive units, lessons, and tasks. (15%) 

■ The Work Students Do: What the students work on and how they 
doit. (25%) 

■ Student Diversity: How the program deals with diverse cultural and 
economic backgrounds, achievement levels, English language 
proficiencies, and interests of students. (10%) 

■ Assessments: How the program integrates assessment with instruc- 
tion, and how the program helps teachers assess student perfor- 
mance. (10%) 

■ Support for the Teacher: How the resources support what the 
teacher does in the classroom. (20%) 
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Each category offers a different perspective of how the total program is 
intended to be experienced by the students. In using these categories, 
reviewers should keep in mind the following general points: 

■ These categories are not distinct; they overlap. For example, students’ 
experiences in a program cannot be judged simply by looking at the 
category, “The Work Students Do.” Their experiences will also be 
affected by the role of the teacher, by the kinds of units and tasks that 
students work on, and by the mathematical content of the program. 

■ In each category, all of the components of an instructional program 
(such as student resources, teacher resources, and technology) are to be 
examined in terms of how they work together to provide a quality 
program for students in a classroom. These criteria do not presuppose 
the presence or absence of any particular component. It is possible to 
design a complete program that does not have a single student text, 
and it is possible to design a high-quality program that does have a 
student text at its center. Similarly, videotapes, computer software, and 
other technology might or might not be included in a program. 

■ Each category is described by a few paragraphs. These paragraphs 
and the subpoints within them are not necessarily of equal weight. 

They should not be judged individually. Rather, they should be used 
as an aid to identifying the qualities that contribute to a category. 

■ Instructional resources need to be descriptive enough to help consci- 
entious teachers implement a new kind of program, yet not so dghdy 
structured that teachers have litde flexibility in their implementation. 

■ In applying these criteria, reviewers will take into account appropri- 
ateness for particular grade levels. 

A rating, ranging from 0 to 5, is made for each category. Paragraph 
descriptions of “5,” “3.5,” and “1” ratings are provided in this evaluation 
form to assist the reviewer. A weighted score of 70 out of a possible 100 
(average rating 3.5) will be required for IREP recommendation for adoption. 
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The following notes supplement the tables that follow: 

* The strands as defined in the Mathematics Framework (logic and 

language, geometry, functions, discrete mathematics, measurement, 
number and operation, algebra, and statistics and probability) and 
the Curriculum and Evaluation Standards for School Mathematics of the 
Nadonal Council of Teachers of Mathematics, are specified by grade 
spans, not by grade levels. The levels of analysis will be at those of 
the 13 standards, for which the bullets serve as exemplars of what 
the standards look like in that span. It is permissible to introduce an 
idea from a later span at an earlier time and to condnue to study an 
idea from an earlier span at a later time. It is also permissible to 
include ideas that are not specifically mentioned in the Standards . 

** Definition of “all three areas”: 

■ Basic Skills are those skills that every student needs to be able to 
use mathematics and solve problems. Basic skills are not only part 
of the number and operadon strand, but are also part of all other 
strands. For example, students need to know: number facts, how to 
find correct answers to addition, subtraction, muldplication, and 
division problems; fractions, decimal, and percent equivalencies; 
how to measure; correct terms for geometric shapes; how to read 
graphs of data; and how to solve equations. 

■ Conceptual understanding means that students make sense of the 
mathematics operations they perform. They not only know hotv to 
apply skills but also when to apply them and why they are being 
applied. Conceptual understanding provides students with the basis 
for seeing the reladonships between skills and problem solving and 
among mathematical ideas. Students with conceptual understand- 
ing see the structure and logic of mathematics, use mathematics 
more flexibly and appropriately, and are able to recall or adapt rules 
because they see the larger pattern. 

■ Problem solving requires the use of mathematical reasoning — analyzing 
the situation, thinking about approaches that are reasonable, considering 
appropriate methods to find a solution, and determining how to verify 
that results make sense in the context of the original situation. Solving a 
problem involves applying mathematical skills, understandings, and 
experiences to resolve new or perplexing situations. 

ITS 
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Evaluation Table: Mathematical Content 



Mathematical Content (5) Mathematical Content (3.5) I Mathematical Content (1) 



The program reflects a balance of basic skills, 
conceptual understanding, and problem solv- 
ing, within the eight strands* included in the 
1 992 Mathematics Framework. 



At each grade level, all three areas** (basic 
skills, conceptual understanding, and problem 
solving) are emphasized; none is neglected nor 
deemphasized more than another, and overall, 
the program reinforces the relationships among 
the areas. 



At each grade level, basic skills, conceptual un- 
derstanding, and problem solving are devel- 
oped in most, if not all, of the strands,* and all 
of the strands are developed in each grade 
span. Most, if not all, of the strands* are devel- 
oped in each year's program. Overall, the units 
of instruction interweave ideas from more than 
one strand. The unifying ideas are fully ex- 
plored over the course of a given year. The 
mathematical subject matter is coherent. 



''Number sense" is usually developed in realistic 
contexts, and students nearly always produce 
numerical results for a purpose. Students have 
ample opportunities to devise their own proce- 
dures and are consistently expected to decide 
the most efficient means for calculating a nu- 
merical result in a given situation: mentally, with 
paper and pencil, or with a calculator. The 
numbers in problems are nearly always realistic, 
and problems in upper elementary grades and 
beyond have rarely been contrived to keep so- 
lutions confined to integers. 



Most of the program reflects a balance of basic 
skills, conceptual understanding, and problem 
solving, within the eight strands* included in 
the 1 992 Mathematics Framework . 



At each grade level, all three areas** (basic skills, 
conceptual understanding, and problem solving) 
are included, although one area may be empha- 
sized more than the others. Frequently, lessons 
includeall threeareas, and overall, the program 
reinforces the relationships among the areas. 



At each grade level, basic skills, conceptual un- 
derstanding, and problem solving are devel- 
oped in most of the strands,* and most of the 
strands are developed in each grade span. Most 
of the strands* are developed in each year's 
program. More than half of the units of instruc- 
tion interweave ideas from more than one 
strand, while others concentrate on a single 
strand. The unifying ideas are explored over 
the course of a given year, though a few may be 
given limited attention. The mathematical sub- 
ject matter in a few of the units fails to meet the 
criteria of coherence. 



More than half of the time, "number sense" is 
developed in realistic contexts, and students 
produce numerical results for a purpose. Stu- 
dents have regular opportunities to devise their 
own procedures and to decide the most effi- 
cient means for calculating a numerical result 
in a given situation: mentally, with paper and 
pencil, or with a calculator. The numbers in 
problems are usually realistic, and problems in 
upper elementary grades and beyond are usu- 
ally not contrived to keep solutions confined to 
integers. 



The program does not reflect a balance of basic 
skills, conceptual understanding, and problem 
solving, within the eight strands* included in 
the 1 992 Mathematics Framework. 



At some grade levels, one of the three areas** 
(basic skills, conceptual understanding, and 
problem solving) may be omitted, and/or one 
area may be emphasized at the exclusion of 
the others. Frequently, lessons address only 
one of the three areas. 



Some of the strands* are ignored or treated 
superficially in each year's program. Units of 
instruction typically treat one strand indepen- 
dent of the others. The unifying ideas are 
treated superficially, if at all. The mathemati- 
cal subject matter is fragmented and fails to 
achieve coherence. 



"Number sense" is not developed in realistic 
contexts; students often produce numerical re- 
sults without purpose and have few opportuni- 
ties to devise their own procedures. They are 
usually required to decide the most efficient 
means for calculating a numerical result in a 
given situation: mentally, with paper and pencil, 
or with a calculator. The numbers in problems 
are often unrealistic, and problems in upper el- 
ementary grades and beyond have often been 
contrived to keep solutions confined to integers. 
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Evaluation Table: Program Organization and Structure 



Program Organization and Structure (5) | Program Organization and Structure (3.5) | Program Organization and Structure (1 ) 



The instructional resources reflect a balance of 
basic skills, conceptual understanding, and 
problemsolving. 

The instructional resources present students 
with coherent, connected, and accessible 
mathematical experiences organized into 
units of work lasting one to six weeks that 
have instructional and mathematical coher- 
ence. While a year's work may include some 
activities that do not belong to any unit or 
units that are distributed and interspersed 
over a period of time, most of the year's work 
is organized in concentrated units. 



Units include investigations, problems and exer- 
cises that are related to one another. Assign- 
ments vary in length. Some are relatively large 
and complex and may form the backbone of 
the unit, while others may be more limited in 
their scope and provide tne tools or techniques 
for successful completion of the investigation. 
Lessons and tasks within a unit most often in- 
volve concepts from more than one strand and 
explore their interconnections and relationships. 
Some tasks are quantitative, interdisciplinary 
"real life" problems; others are more purely 
mathematical investigations, including games 
and puzzles. 



The instructional resources outline a default 
sequence for the units and indicate how teach- 
ers can use the program flexibly, rearranging, 
substituting, or modifying units and tasks to 
meet the needs and interests of their students. 
Students are given opportunities to decide 
which tasks to work on or in which order, such 
as provided in a "menu" format or alternative 
investigations. 



From year to year, the units — and the tasks 
within them — increase in depth and complex- 
ity. Students encounter the same unifying 
mathematical ideas in different or more com- 
plex context. Gradually, tasks increase in du- 
ration; tasks include more abstraction and 
formalism; students grapple with increasingly 
complex questions and investigations; and 
students assume more and more responsibil- 
ity for developing complete and comprehen- 
sive reports or products. 



Most of the instructional resources reflect a 
balance of basic skills, conceptual under- 
standing, and problem solving. 

Most of the year's work consists of units that 
contain investigations, problems, and exer- 
cises that are related to one another, though a 
few of the connections may be weak. Assign- 
ments of varying length are found in most 
units; while there are substantial investiga- 
tions in some units, others may contain only 
brief or introductory investigations. More of- 
ten than not, units involve concepts from 
more than one strand. 



The instructional resources outline a default 
sequence for the units, with some general 
suggestions of ways teachers can modify the 
program to meet the needs and interests of 
their students. Students are occasionally 
given opportunities to decide which tasks to 
work on or in which order, such as provided 
in a "menu" format. 



From year to year, a few units may be at a level 
of depth similar to the previous year's, but 
most of the units — and the tasks within them — 
increase in depth and complexity and chal- 
lenge students to assume increasing 
responsibility for developing complete reports 
or products. 



The instructional resources do not reflect a 
balance of basic skills, conceptual under- 
standing, and problem solving. 

The year's work consists of activities that are 
fragmented or disconnected from one another. 
Connections within a unit — investigations, 
problems, and exercises — are often weak or 
contrived. Assignments are frequently narrow, 
with a few substantial investigations and many 
one-day lessons. Units often focus on a single 
strand or set of procedures, or fail to demon- 
strate mathematical ideas in many settings. 



The instructional resources outline a fairly 
rigid sequence for the units, with few sugges- 
tions of ways teachers can modify the pro- 
gram to meet the needs and interests of their 
students. Students are rarely given opportuni- 
ties to decide which tasks to work on or in 
which order, such as provided in a "menu" 
format. 



From year to year, there is little progression in 
the level of depth, contexts are repetitious, 
and few units challenge students to assume 
responsibility for developing complete reports 
or products. 
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Evaluation Table: The Work Students Do 



The Work Students Do (5) 



Students are consistently expected to think 
and reason in their mathematical work, to 
work on a variety of challenging and mean- 
ingful mathematical tasks, and to conjec- 
ture and pursue possibilities without 
knowing the answer will follow. Most as- 
signments ask for complete student work; 
students are asked to think and communi- 
cate, to draw on mathematical ideas, and 
to use mathematical tools and techniques 
effectively. 



Students encounter a varied program of ex- 
ercises, problems, and investigations that re- 
quires the use of a balance of basic skills, 
conceptual understanding, and problem 
solving and that includes all of tne strands 
and unifying ideas. Most assignments are 
open ended and allow multiple approaches. 
Students are often asked to formulate math- 
ematical questions, generally choose the ap- 
proaches to take, frequently reflect on the 
work they are doing, and make connections 
among the mathematical ideas. Many tasks 
required time and deliberation and, espe- 
cially after third grade, are continued over 
several days. On these extended assign- 
ments, resources help teachers set a clear 
standard of performance for student work 
and suggest ways to help students meet the 
standard. 



Students are asked to work individually and 
interact with one another. They often work 
in small heterogeneous groups to communi- 
cate their findings, orally or in writing. 

When working together, they are expected 
to share approaches, conjectures, difficul- 
ties, results, and evidence within their group 
and with other groups. Students are often 
asked to explore situations, gather data, and 
present their conclusions to other audi- 
ences, including interacting with members 
of their families in homework assignments. 



Students are directed to use manipulative 
resources and technology to explore ideas 
and solve problems. It is assumed that a 
variety of tools is continually available for 
students to use and that every student has 
access to a calculator at all times and may 
choose to use it for any occasion except 
when the purpose is developing mental 
dexterity. 



The Work Students Do (3.5) 



More often than not, students are ex- 
pected to think and reason in their math- 
ematical work, to work on a variety of 
challenging and meaningful mathematical 
tasks, and to conjecture and pursue possi- 
bilities without knowing the answer will 
follow. Many assignments ask for com- 
plete student work; students are asked to 
think and communicate, to draw on math- 
ematical ideas, and to use mathematical 
tools and techniques effectively. 



All students encounter a varied program, 
including experience with exercises, prob- 
lems, and investigations, that frequently re- 
quires the use of a balance of basic skills, 
conceptual understanding, and problem 
solving and that includes most of the 
strands and unifying ideas. Some assign- 
ments are open ended and allow multiple 
approaches. In these assignments, students 
are asked to formulate mathematical ques- 
tions, to choose the approaches to take, to 
reflect on the work they are doing, and to 
make connections among the mathemati- 
cal ideas. Many tasks require time and de- 
liberation and, especially after third grade, 
are continued over at least a few days. On 
these extended assignments, resources 
help teachers set a clear standard of perfor- 
mance for student work and suggest ways 
to help students meet the standard. 

More than half the time, students are asked 
to interact with one another. They often 
work in small heterogeneous groups to 
communicate their findings, orally or in 
writing. When working together, they are 
expected to share approaches, conjectures, 
difficulties, results, and evidence within 
their group and with other groups. At 
times, they are asked to explore situations, 
gather data, and present their conclusions 
to other audiences, including interacting 
with members of their families in home- 
work assignments. 

Students are often directed to use manipu- 
lative resources and technology to explore 
ideas and solve problems. Students have 
the choice of tools to use, although some of 
the choices may be limited or are made for 
the students. 



rr* 



me. 



T3J 



The Work Students Do (1 ) 



Only occasionally are students expected to think 
and reason in their mathematical work, to work 
on a variety of challenging and meaningful math- 
ematical tasks, and to conjecture and pursue pos- 
sibilities without knowing the answer will follow; 
more often students are expected to follow pre- 
scribed directions to achieve a predetermined an- 
swer. Few assignments ask for complete student 
work in which students are asked to think and 
communicate, to draw on mathematical ideas, 
and to use mathematical tools and techniques ef- 
fectively; many assignments focus on one of the 
aspects of mathematical power independent of 
the others. 



Students do not encounter a varied program; 
some strands and unifying ideas are treated su- 
perficially, if at all, and the program does not 
provide experience with exercises, problems, 
and investigations. Assignments do not re- 
quire the use of a balance of basic skills, con- 
ceptual understanding, and problem solving. 
Few assignments are open ended or allow 
multiple approaches; many are overly directed 
so that students do not have the opportunity to 
formulate mathematical questions, choose the 
approaches to take, reflect on the work they 
are doing, and make connections among the 
mathematical ideas. Few tasks require time 
and deliberations, and most are confined to a 
single day. Resources do not help teachers set 
a clear standard of performance for student 
work or suggest ways to help students meet 
the standard 



Students are infrequently asked to interact 
with one another, and when they are, their 
communication is limited to their results 
and does not usually include their ap- 
proaches, conjectures, difficulties, and evi- 
dence. They are rarely expected to 
communicate their thinking in writing, or to 
communicate to other groups or other audi- 
ences, such as members of their families, in 
homework assignments. 



Students are rarely directed to use manipula- 
tive resources and technology to explore ideas 
and solve problems, and when they do, they 
have a I imited choice of tools to use or ways to 
use them. 
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Evaluation Table: Student Diversity 



Student Diversity (5) 



Student Diversity (3.5) 



Student Diversity (1 ) 



Al I students participate fully in each unit; as- 
sistance provided to students having difficulty 
is in addition to, not instead of, the regular 
program, with the goal of supporting success- 
ful participation in the regular program. The 
tasks and problems students work on are ac- 
cessible to all students. Many problems are 
rich and open and can be investigated at 
many different levels. Many tasks or lessons 
use students' personal, family, or cultural ex- 
periences to create the specific context for 
the lesson and to include parents as partners 
in the process. There is room in each unit for 
some students to pursue depth, complexity, 
or novelty in some aspect of the unit's investi- 
gations, according to their interests or their 
rapid grasp of the ideas. 



The regular program is specifically made ac- 
cessible to English language learners (ELL) by 
providing editions in the five most common 
languages other than English spoken in Cali- 
fornia, comparable in quality to those writ- 
ten in English, or by providing glossaries; 
summaries of key concepts; and directions, 
instructions, and/or problems and tasks in 
these five primary languages. 

The resources provide teachers with general 
advice on all of the following points and fre- 
quent lesson-specific suggestions to support 
the learning of all students, including: 

• ways to encourage students to connect 
lessons and content of lessons to their 
personal, family, or cultural experience; 

• ways to encourage students to value the 
points of view and the experiences of 
others; 

• the use of peer support and collaborative 
learning groups; and 

• how to work with students whose primary 
language is not English, including tech- 
niques for the use of the primary language 
and Specially Designed Academic Instruc- 
tion in English (SDAIE) so that the program 
for these students is not limited or diluted. 
The focus in this criterion is to allow ELL 
(LEP) students accessibility to the program. 



All students participate in each unit. The 
entry point for tasks and problems is acces- 
sible to all students, but there is a gradient 
of difficulty that may sort studentsby their 
speed or verbal fluency. At that point, tem- 
orary assistance is provided to students 
aving difficulty in lieu of keeping them in 
the regular program. Remaining students 
are provided with rich and open activities, 
in contrast to the narrow and mechanical 
activities that may be provided for the stu- 
dents having difficulty. Some tasks or les- 
sons use students' personal, family, or 
cultural experiences to create the specific 
context for the lesson, and most often in- 
clude parents as partners in the process. 
There is room in many units for some stu- 
dents to pursue depth, complexity, or nov- 
elty in some aspect of the unit's 
investigation according to their interests or 
their rapid grasp of the ideas. 



The regular program is made somewhat ac- 
cessible to English language learners (ELL) 
by providing editions in Spanish as well as 
in at least two languages other than English, 
comparable in quality to those written in 
English, or by providing glossaries; summa- 
ries of key concepts; and directions, in- 
structions, and/or problems and tasks in these 
four primary languages. 

The resources provide teachers with gen- 
eral advice on all of the following points, 
and occasional lesson-specific suggestions, 
as applicable, to support the learning of all 
students, including: 

• ways to encourage students to connect 
lessons and content of lessons to their 
personal, family, or cultural experience; 

• ways to encourage students to value the 
points of view and the experiences of oth- 
ers; the use of peer support and collabo- 
rative learning groups; 

• and how to work with students whose 
primary language is not English, including 
techniques Tor the use of the primary lan- 
guage and Specially Designed Academic 
Instruction in English (SDAIE) so that the 
program for these students is not limited 
or diluted. The focus in this criterion is to 
allow ELL (LEP) students accessibility to 
the program. 



All students participate in each unit. The entry 
point for tasks and problems is not always ac- 
cessible to all students, and there is a steep gra- 
dient of difficulty that quickly sorts studentsby 
their speed or verbal fluency. Assistance for stu- 
dents having difficulty keeps them out of the 
regular program and limits their experience to 
narrow and mechanical tasks. Few tasks or les- 
sons use students' persona I, family, or cultural 
experiences to create the specific context for 
the lesson. Parents are not often included as 
partners in the progress. There is little room in 
most units for some students to pursue depth, 
complexity, or novelty in some aspect of the 
unit's investigations, according to their interests 
or their rapid grasp of the ideas. 



The regular program is not generally accessible to 
English language learners (ELL). Directions, in- 
structions, problems and tasks, and/or glossaries 
and summaries are not provided in any language 
other than English. 



The resources provide teachers with general 
advice on all of the following points and a few 
lesson-specific suggestions to support the 
learning of all students, including: 

• ways to encourage students to connect les- 
sons and content of lessons to their per- 
sonal, family, or cultural experience; 

• ways to encourage students to value the 
points of view and the experiences of oth- 
ers; 

• the use of peer support and collaborative 
learning groups; and 

• how to work with students whose primary lan- 
guage is not English, including techniques for 
the use of the primary language and Specially 
Designed Academic Instruction in English 
(SDAIE) so that that the program for these stu- 
dents is not limited or diluted. The focus in this 
criterion is to allow ELL (LEP) students accessi- 
bility to the program. 
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Evaluation Table: Assessment 



Assessment (5) Assessment (3.5) Assessment (I) 



Assessment is consistently integrated into the 
instructional program. The instructional re- 
sources help teachers use assessment in a va- 
riety of ways to get information about what 
the student or groups of students understand 
and are able to do in solving mathematical 
problems. Assessment measures a balance of 
basic skills, conceptual understanding, and 
problem solving within the eight strands of 
mathematics. 



Specific assessment tasks are included in the 
units, though they may or may not be distin- 
guished from learning tasks because they have 
similar characteristics. Specifically, students 
formulate problems, consider ana apply a vari- 
ety of approaches, determine and explain their 
findings; use tools, such as calculators and 
manipulates, and other resources, such as 
notes or reference materials; and have ample 
time to work on assessment tasks to revise and 
resubmit important assignments to bring per- 
formance up to high-quality standards. 



The resources include general suggestions to 

the teacher concerning the following, with 

frequent unit-specific suggestions: 

• how to use learning tasks for assessment 
purposes; 

• how to observe, listen to, and question stu- 
dents while they work, as well as how 
teachers might keep track of insights they 
may have about the students; 

• how to organize and use student portfolios; 

• how teachers can keep parents informed 
about the progress of their children and 
about the variety of assessment methods be- 
ing used: both what they are and why they 
are important; and 

• how to involve students in self-assessment. 



Assessment is generally integrated with the in- 
structional program. The instructional re- 
sources help teachers use assessment in a 
variety of ways to get information about what 
the student or groups of students understand 
and are able to do in solving mathematical 
problems. Assessment frequently measures a 
balance of basic skills, conceptual under- 
standing, and problem solving within the 
eight strands of mathematics. 



Specific assessment tasks are included in the 
units and display most of the following desir- 
able characteristics: students formulate prob- 
lems, consider and apply a variety of 
approaches, determine and explain their find- 
ings; use tools, such as calculators and 
manipulatives, and other resources, such as 
notes or reference materials; and have ample 
time to work on assessment tasks and fre- 
quently have the opportunity to revise and 
resubmit important assignments to bring per- 
formance up to high-quality standards. 



The resources include general suggestions to 

the teacher concerning the following, with 

occasional unit-specific suggestions: 

• how to use learning tasks for assessment 
purposes; 

• how to observe, listen to, and question stu- 
dents while they work, as well as how 
teachers might keep track of insights they 
may have about the students; 

• how .to organize and use student portfolios; 

• how teachers can keep parents informed 
about the progress of their children and 
about the variety of assessment methods be- 
ing used: both what they are and why they 
are important; and 

• how to involve students in self- assessment. 



Assessment is not integrated with the instruc- 
tional program. The instructional resources 
do, however, help teachers use assessment 
to get information about what the student or 
groups of students understand and are able 
to do in solving mathematical problems. As- 
sessment does not measure a balance of ba- 
sic skills, conceptual understanding, and 
problem solving within the eight strands of 
mathematics. 



Specific assessment tasks are included in the 
units. They display only a few of the follow- 
ing desirable characteristics: students for- 
mulate problems, consider and apply a 
variety of approaches, determine and ex- 
plain their findings; use tools, such as calcu- 
lators and manipulatives, and other 
resources, such as notes or reference mate- 
rials; and have ample time to work on as- 
sessment tasks and frequently have the 
opportunity to revise and resubmit impor- 
tant assignments to bring performance up to 
high-quality standards. 



The resources include general suggestions to 

the teacher concerning the following, with 

few unit-specific suggestions: 

• how to use learning tasks for assessment 
purposes; 

• how to observe, listen to, and question stu- 
dents while they work, as well as how 
teachers might keep track of insights they 
may have about the students; 

• how to organize and use student portfolios; 

• how teachers can keep parents informed 
about the progress of their children and 
about the variety of assessment methods 
being used: both what they are and why 
they are important; and 

• howto involve students in self-assessment. 
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Evaluation Table: Support for the Teacher 



Support for the Teacher (5) I Support for the Teacher (3.5) Support for the Teacher (1) 



The resources provide many lesson- and unit- 
specific suggestions and illustrative examples of 
how the teacher can instruct and facilitate the 
student behaviors identified under "The Work 
Students Do." The resources include the fol- 
lowing: 

• description of the important mathematical 
ideas in the units, including how students are 
to encounter the mathematical ideas and 
how the experiences are related to what is 
known about children's learning or develop- 
mental level; 

• description/pictures of what units and les- 
sons will look like when implemented in the 
classroom; 

• information about what is important to do and 
say in a lesson or unit; 

• suggestions for questions to ask and ways to 
respond that keep students' thinking open 
and help them on what they have done, 
how teachers might think about and reflect 
on what happens in a lesson or unit, when 

it makes sense to present information to stu- 
dents and how to do it; 

• suggestions for working with a diverse class- 
room of students, helping students work to- 
gether productively, as well as managing 
manipulative materials, calculators, comput- 
ers, and other tools so they are accessible 
when students want to use them; 

• suggestions for helping students communicate 
more effectively about their mathematical 
thinking; 

• suggestions for involving parents and keep- 
ing them informed about the program; and 

• suggestions for assessing student performance. 



The resources provide information, suggestions, 
and some illustrative 

examples of how the teacher can instruct and 
facilitate the student behaviors identified under 
"The Work Students Do" The materials in- 
cl ude most of the fol lowi ng: 

• description of the important mathematical 
ideas in the units, including how students are 
to encounter the mathematical ideas and 
how the experiences are related to what is 
known about children's learning or develop- 
mental level; 

• description/pictures of what units and les- 
sons will look like when implemented in the 
classroom; 

• information about what is important to do and 
say in a lesson or unit; 

• suggestions for questions to ask and ways to 
respond that keep students' thinking open 
and help them reflect on what they nave 
done, how teachers might think about and 
reflect on what happens in a lesson or unit, 
when it makes sense to present information 
to students and how to do it; 

• suggestions for working with a diverse class- 
room of students, helping students work to- 
gether productively, as well as managing 
manipulative materials, calculators, comput- 
ers, and other tools so they are accessible 
when the students want to use them; 

• suggestions for helping students communicate 
more effectively about their mathematical 
thinking; 

• suggestions for involving parents and keep- 
ing them informed about the program; and 

• suggestions for assessing student performance. 



The resources provide limited information, 
suggestions, and few illustrative examples of 
how the teacher can instruct and facilitate the 
student behaviors identified under "The 
Work Students Do." Many of the suggestions 
are general rather than lesson or unit specific 
in nature. The materials include few of the 
following: 

• description of the important mathematical 
ideas in the units, including how students 
are to encounter the mathematical ideas 
and how the experiences are related to 
what is known about children's learning or 
developmental level; 

• description/pictures of what units and les- 
sons will look like when implemented in the 
classroom; 

• information about what is important to do and 
say in a lesson or unit; 

• suggestions for questions to ask and ways to 
respond that keep students' thinking open 
and help them reflect on what they nave 
done, how teachers might think about and 
reflect on what happens in a lesson or unit, 
when it makes sense to present information 
to students and how to do it; 

• suggestions for working with a diverse class- 
room of students, helping students work to- 
gether productively, as well as managing 
manipulative materials, calculators, comput- 
ers, and other tools so they are accessible 
when the students want to use them; 

• suggestions for helping students communicate 
more effectively about their mathematical 
thinking; 

• suggestions for involving parents and keep- 
ing them informed about the program; and 

• suggestions for assessing student performance. 
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K-8 Mathematics Instructional Resources Evaluation Form 

Publisher 

Title 

Grade Span 



Cateeorv 


Rating 


Weierht 




Total 


Mathematical Content 




x 4 


= 


Program Organization 
and Structure 




x 3 




The Work Students Do 




x 5 




Student Diversity 
Assessment 




x 2 






x 2 




Support for the Teacher 




x 4 
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Council of Chief State School 
Officers (CCSSO) State Curriculum 
Frameworks and Standards Map: 
Definitions of Categories and Concepts 
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1 . Subject(s) of Framework 

A. Title : Official and complete title of framework or standards document(s) 

B. Date of document reviewed : 



C. Status of framework : Classify the current status in terms of one of the 
following three phases:. (Select one) 

■ Developing or working draft: Document is in a predistribution and 
development phase; it is not yet disseminated for formal review 
beyond a limited circle of writers or experts. 

■ Review draft: Document is a completed draft that is being formally 
disseminated for comment and final revisions. 

■ Adopted: Document serves as official guidance, mandate, or policy. 

D. Implemented/Disseminated : 

E. Planned revision date : 



F. Main components of document : 

(Components present — example list) 

Content standards: Content knowledge and skills students are 
expected to know and be able to do in mathematics or science, often 
by grade level. 

Performance standards (or outcomes): Description of the levels of 
proficiency to which content standards shall be attained by students. 
Pedagogy/Instructional strategies: Pedagogical approaches, ideas, 
procedures, and processes of teaching recommended for use with 
curriculum. 
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■ Linking to policies and enabling conditions: Policies or guidelines 
for assessments, professional development, curriculum design and 
organization, materials and texts, school and system support, tech- 
nology, and facilides. 

■ Implementation plan for state: 

■ Assessment plans, examples: 

G. Single vs. muldple subjects : Framework document includes one subject or 
multiple subjects. 

H. Related documents : 

■ Key state-designed and disseminated documents, in addition to the 
framework, related to the curriculum framework or standards 
(referenced or identified from other information). Indicate 
author (s), title, and publication date. 

2. State Context 

A. State's vision for mathematics and science : 

■ Specific to one subject vs. broad, general. 

■ Contains specific information about needs, interests of state vs. 
general statement of vision. 

■ Page number for vision statement. 

B. Intent or purpose of the framework/standards : Indicate which of the 
following concepts best describe the rationale , goals, or purpose, as stated in the frame- 
work; add any concepts the state identifies that are not described by these overarching 
statements. 

■ Develop student knowledge and habits of mind. 

■ Modernize and change instruction. 

■ Design recommended curricula and instructional resources. 

■ Ensure equity and access for all students. 

■ Make or implement assessment/ evaluation policy. 

■ Involve parents, business, and other community members. 



Other. 
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C. Major sources : Major national documents or reports , especially: National 
Council of Teachers of Mathematics' Curriculum and Evaluation Standards for 
School Mathematics, Teaching Standards, Assessment Standards; National 
Academy of Sciences /National Research Council National Science Education Stan- 
dards; American Association for the Advancement of Science Benchmarks; other 
states' frametvorks or reports; or other studies or reports referenced in the opening 
sections of the framework that establish the vision and define the conceptual foundation 
for science /mathematics education. 

■ Reference page numbers for national documents. 

■ Characterize how they are used or modeled closely after national 
documents; frequent references; listed as resource. 

■ List other states that are referenced as sources. 

D. Mandates or state initiatives supporting frameworks and standards : 
Major funding initiatives (internal or external , regulatory or advisory ), supplemental 
financial resources , legislative or board-sponsored mandates , requirements , and 
supportive arrangements that advanced or contributed to the development and/or 
implementation of the framework. Mention here any specific decisions that enhance 
local resources; technology; use of school time; professional development; facilities; 
schedules , and so forth. 

■ Legislative or state board mandates, rules, or regulations. 

■ External funding — from federal, non-profit agencies, and so forth. 

(List the funding sources/ agencies) . 

E. Target audiences : The key groups that the framework zoas designed to 
inform, assist , or advise , particularly those expected to make most immediate and 
regular use of the frametvork to carry out their mission and primary responsibilities. 

■ Teachers: Teachers in all subjects and for all grades or special 
groups; and certified support personnel such as media specialists, 
counselors, and so forth. 

■ School and district curriculum planners: Curriculum and instruc- 
tional supervisors, designers, or writers. 
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■ School administrators: Administrators at school site and central 
office levels. 

■ Assessment developers: Test or item authors, contractors designing 
test and item specifications, administrators or assessment managers. 

■ Board members: Members of district school boards and/or school- 
site boards or management teams. 

■ Parents: PTA organizations, individual parents, and other parent 
advisory or participant groups. 

■ Business partners: Business or industry partners who work in a 
formal relationship with schools or school boards. 

■ Other community constituents: Public sector agencies and other 
parent or public groups working with the school system. 

■ Other: Indicate other users/audiences not indicated above. 



F. Key development activities : Major steps that best describe phases , events , 

or processes that were used in completing the current edition of the frameiuork if the 

development process is described in the document. 

■ Expert panel (s) and/or advisory committee (s). 

■ Reviews and analyses of other similar or model documents. 

■ Suggestions, ideas, and exemplars from teachers, educators, and 
subject specialists. 

■ Teacher participation in writing and editing for the framework’s text 
(including writing of suggested standards). 

■ Successive drafts were written, reviewed, and modified. 

■ Focus groups for review/discussion of drafts. 

■ Pilot study in districts and/or classrooms. 

■ Wide dissemination within schools and throughout communities and 
the state. 

■ Other (indicate other activities that differ from those listed above). 
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3. Standards 

A. Content standards : Categories or headings that organize subject content 
and indicate the term(s) used for the categories , for example , strands, standards, 
themes, disciplines, and so forth. List only the broad outline; include examples of detail 
in the sample Standards in the next section. Describe the state's definition and uses of 
content or performance standards as specified in the framezoork document. 

■ Definition: Quote or paraphrase, page reference. 

■ Number of standards: 

■ Grade levels/clusters of standards: If content categories differ 
greatly across grade levels, separately list content clusters for each 
grade level or group. 

Supply example of content standard for subject area. 

B. Benchmarks/Indicators : Next level of detail beloiv Standards. 

■ Definition: Term used for this level. 

■ Number of benchmarks per Standard. 

■ Grade levels/clusters of this level. 

Supply example of a benchmark that matches the content standard in- 
cluded above. 

C. Cross-cutUng themes : Concepts for organizing and implementing curricu- 
lum content that transcend framexuork’s major content organizers, including processes, 
themes, attitudes, habits of mind, and so forth. They appear repeatedly across several 
components or sections of the frameiuork, in various content areas and topics. 

4. Performance Standards 

A. State's definition and term for performance standards : Where they are 
found and name of document. Hozu they are related to content standards. 

■ Short quote on definition, page number. 

■ Number of performance standards; grade clusters. 

Supply example of performance standard that matches content standard 
defined above. 
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B. Performance standards are designed and described to be used for 
developing : 

■ External accountability/state assessment. 

■ End-of-course assessment. 

■ Student accountability/ classroom assessment. 

Describe use(s) for state from interviews. 

5. Assessment 

How assessment is related to standards: 

■ Classroom or instructional assessment. 

■ Accountability. 

■ Examples of assessments and page numbers. 

■ Types of exercises/ tasks described or modeled. 

6. Features 

Methods of presenting and communicating content used regularly and through- 
out the document to illustrate the recommended curriculum content or pedagogical 
practices. (Indicate page numbers for good examples.) 

■ Vignettes: Illustrative classroom scenarios or essays that describe teaching , 
situating instruction in a context that demonstrates the give and take among 
children and their teachers and shows with considerable descriptive detail 
how teachers define in practice specific content areas and instructional 
approaches. Vignettes often include dialogue. 

■ Models or examples of instructional practice: u Hotu to” examples of 
techniques or thumbnail sketches of practice. Compared with vignettes , models 
are shorter and more static , and they do not attempt to reflect classroom context , 
discourse ; or exchanges that occur among teachers and students. 

■ Activities and/or instructional tasks: Lists or outlines of activities 
or tasks , reported with a minimum of detail , surrounding context , give or 
take , or dialogue. 

■ Diagrams, graphics, photographs: Visuals used to model or illustrate 
the instructional activities , tools , structures , patterns , or approaches. 

■ Other unique features: (ways of presenting content , etc.) 
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7. Linkages to Policies and Enabling Conditions 

Linkages of content to policies and recommended enabling conditions : 
Extent of alignment of framework's vision , content , standards , and pedagogy with 
other state education policies , or with policy decisions that are left to district or school 
levels; policies that enable teachers and schools to improve mathematics/ science. Record 
the linkages within one of the following two dimensions: 

■ Current and existing policy links are reported within the document. 

■ Policy linkages are proposed and/or recommended, but they are not 
yet regulated by statute or rule. 



Framework/Standards Linkages 
(Content links to following) 


Existing links reported in 
document 


Links are recommended 


Professional development 






Assessment for external 
accountability 






Selecting, developing, or using 
materials and textbooks 






Teacher preparation and certification 






Student support services 






School and curriculum organization, 
decision making, uses of time, 
district and school support 






Technology integration 






Involving community 
representatives, parents, and 
business leaders in planning, review, 
and policy-making committees 






Facilities 







Other documents providing linkages to content standards. 
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8. Equity 

Indicate the ways the framework document addresses race /ethnicity, gender equity i, 

inclusiveness , and attentiveness to handicapping conditions in the curriculum. Frame- 
work clearly describes or illustrates state recommendations for any one equity element 

that is checked. 

■ Where/in which sections is equity treated? 

■ How is it treated? Cite examples and page numbers. 

■ List examples of where equity is treated. 

■ Rationale and vision statements: (e.g, importance of all students 
learning mathematics /science in K-12 , how framezvork will aid this goal.) 

■ Vignettes, activities, and sample problems: (Examples are multira- 
cial/ethnic and cross-gender; refer to education for culturally diverse, handi- 
capped, and limited-English-speaking students; and describe diverse teaching 
and learning situations.) 

■ Instructional strategies, curriculum organization: (e.g, including 

content and instructional techniques that nurture the needs of students from 
varied backgrounds, experiences, cultures, and communities.) 

■ Curriculum organization and course structure; ( e.g., detracking 
curriculum; integrated courses and teacher assignments involving 
multicultural communities in teaching and learning. ) 

■ Materials selection and distribution: (e.g., guidelines for materials 

selection reflect high standards for all groups.) 

■ Assessment approaches: (e.g., strategies that are sensitive to racial/ 
ethnic and gender differences , minimize bias, and maximize opportunities for 
students to demonstrate their knowledge in various ways.) 

■ Staff development: (e.g., proposes strategies to reach educators with fere 
development opportunities.) 

■ Teacher preparation: (e.g, designs that prepare teachers to teach math- 
ematics/science content to diverse student groups.) 

■ Hiring practices: (examples of incentives for increasing numbers of 
minority teachers and administrators.) 

■ Community and business involvement: (e.g., suggested mechanisms 

for increasing the involvement of diverse groups.) 

■ Policies and recommendations: (e.g, state policies that promote equity 
and inclusiveness in teaching, curriculum, assessment, and hiring. ) 
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9. Pedagogy 

The approach to teaching and learning — the instructional strategies, processes, 
or activities recommended, proposed, or modeled — and the means by which these ideas 
are presented . Pedagogy categories are based on the NCTM Curriculum and Evalu- 
ation Standards for School Mathematics (1989) and on the AAAS Science for 
All Americans (1989). 

A. Explanation of Columns : 

■ Lists: Items are listed or outlined with little or no detail about how peda- 
gogical elements are to be used within classrooms or by teachers. 

■ Vignettes and examples: Illustrative stories with specific examples, “ how 
to ” statements describing a story line, or portraits exploring the discourse and 
instructional processes among students and their teachers. These may include 
dialogue, thinking, and reflective practice by both teachers and students. 

■ Context: In many places throughout the frameioork, pedagogy is demon- 
strated through lists, diagrams, pictures, and/ or portraiture that tell the tale 
of events and processes in action. 



Pedagogical Practices 


Lists 


Vignettes and Examples 


Context 


Constructive and active lessons 








Technological applications 








Multiple assessment strategies 








Discourse, dialogue, and argument 








Flexible uses of time and facilities 








Working with a mix of tools, 
manipulatives, textbooks, and re- 
sources 








Experiments and multiple solutions 








Representation, including writing, 
mapping, diagramming, graphing, and 
so forth 
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B. Definitions for pedagogical practices : The framework suggests, models, 

or describes the following pedagogical practices: 

■ Constructive and active lessons: Students actively engage with content, 

materials, tools, and/or peers to build understanding of mathematical and 
scientific ideas, solutions, or explanations. 

■ Technological applications: Students use various technological tools 
(calculators, including those that graph, and computers ) to derive math- 
ematical and scientific solutions. 

■ Multiple assessment strategies: Teacher observations, conferences, self- 

assessments, student journals; projects, experiments, constructions; writing, 
tasks, and real-world problems to solve; formal and informal tests; oral 
presentations; and so forth. 

■ Discourse, dialogue, and argument: Verbal representations of the logic 

underlying mathematical and scientific ideas; strategies call upon students to 
discuss, prove their point of vierv, probe one another’s as well as the teacher’s 
thoughts; and so forth. 

■ Flexible uses of time and facilities: Varying time blocks, classroom 
spaces, libraries, laboratories, and so forth are suggested as resources for 
learning mathematics and science. 

■ Working with a mix of tools, manipulatives, textbooks, and re- 
sources: Activities and tasks suggest ways to involve students in a wide 
range of mathematical and scientific problem-solving tools and materials. 

■ Experiments and multiple solutions: Tasks demonstrate ways that 

mathematics and science are incorporated into applied contexts through 
problem solving, experiments, and experiences that lend themselves to 
alternative solutions with no single “best” answer or solution strategy. 

■ Representation, including writing, mapping, diagramming, graph- 
ing, and so forth: Students are encouraged to explain mathematics and 
science in various formats, including pictures, diagrams, spreadsheets, 
models, maps, and so forth, as well as verbally. 

10. Other Framework Components 

Major sections or chapters of state framework not addressed by preced- 
ing categories. 
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Executive Summary of 

A Splintered Vision: An Investigation 

of U.S. Science and Mathematics Education 

THE SPLINTERED VISION: AN OVERVIEW 



There is no one at the helm of mathematics and science education in 
the United States; in truth, there is no identifiable helm. No single coherent 
vision of how to educate today’s children dominates U.S. educational practice 
in either subject, nor is there a single, commonly accepted place to turn to for 
such visions. Our visions — to the extent that they exist at all — are multiple. 

These splintered visions produce unfocused curricula and textbooks that 
fail to define clearly what is intended to be taught. They influence teachers to 
implement diffuse learning goals in their classrooms. They emphasize famil- 
iarity with many topics rather than concentrated attention to a few. And they 
likely lower the academic performance of students who spend years in such a 
learning environment. Our curricula, textbooks, and teaching all are “a mile 
wide and an inch deep.” 

This preoccupation with breadth rather than depth, with quantity rather 
than quality, probably affects how well U.S. students perform in relation to their 
counterparts in other countries. It thus determines who are our international 
“peers” and raises the question of whether these are the peers that we want to 
have. In today’s technologically oriented global society, where knowledge of 
mathematics and science is important for workers, citizens, and individuals alike, 
an important question is: What can be done to bring about a more coherent 
vision and thereby improve mathematics and science education? 

Reforms have already been proposed by political, business, educational 
and other leaders. Extensive efforts are underway to implement these stan- 
dards, but the implementation process itself is shaped by the prevailing 
culture of inclusion. Like the developers of curricula and the publishers of 
textbooks, teachers add reform ideas to their pedagogical quivers without 
asking what should be taken away. 

The study summarized below represents an effort to describe the nature 
of the diffuse vision of mathematics and science education in the United 
States and to raise questions relevant to policy making. 
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Purpose of A Splintered Vision 

A Splintered Vision (written by William Schmidt, Curtis McKnight, and 
Senta Raizen of the U.S. NaUonal Research Center for the Third Interna- 
tional Mathematics and Science Study and published by Kluwer Academic 
Publishers) discusses data from the analysis of 491 curriculum guides and 628 
textbooks from around the world as part of the recendy completed Third 
International Mathematics and Science Study (TIMSS). It also presents 
detailed accompanying data on teacher practices in the United States and 
two other countries: Germany and Japan. 

The TIMSS is a large-scale, cross-naUonal comparative study of the 
nadonal educational systems and their outputs in about 50 countries. Re- 
searchers examined mathematics and the sciences curricula, instructional 
practices, and school and social factors, as well as conducting achievement 
tesdng of students. They collected data from representative documents that 
laid out official curricular intentions and plans, analyzed entire mathematics 
and science textbooks, and searched entire K-12 textbook series for selected 
“in-depth” topics (subareas within the subject matter.) In six countries 
TIMSS conducted classroom observations, teacher interviewing, and video- 
taping. 

The TIMSS curriculum and teacher data are extensive and cannot be 
explored in a single report. The results of analyses of these data are being 
reported in a series of volumes, three of which are now available. 1 

The present report intends to document and characterize the state of 
U.S. mathematics and science curricula, textbooks, and teaching practices 
and place them in a cross-national context. Unfortunately, this study could 
provide only a snapshot of the “moving target” that is educational practice in 
the United States These data were collected in 1992-93, when the math- 
ematics standards had existed for only three years and the science standards 
were not finalized. 2 The intervening years have been a time of change for 
state curriculum standards and textbooks. The TIMSS data on teacher prac- 
tices discussed here were collected in 1995. 

This report is meant to be descriptive and, to a lesser extent, interpre- 
tive. It is not a plea for specific reforms. We seek to provide data germane to 
the ongoing public debate about science and mathematics education policies 
in the United States. 
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Unfocused Curricula 

Curricula in both mathematics and science in U.S. schools are unfocused 
in comparison with those in other countries studied. The lack of curricular 
focus is more true in mathematics than in science, though physical science 
guides closely resemble mathemadcs in their fragmentation. U.S. curricula 
are unfocused in several respects: 

Topics Covered 

Mathematics curricula in the United States consistently cover far more 
topics than is typical in other countries. The number of mathematics topics in 
the U.S. composite 3 is higher than the 75th percentile internadonally in all 
grades until ninth, when schools typically teach specific courses such as 
algebra, geometry, etc. 

In science, the tendency toward inclusion is similar, though less 
pronounced. The number of science topics in the U.S. composite exceeds the 
50th percentile internationally in all but one grade until the tenth, when 
schools tend to abandon general science approaches for specific courses, such 
as chemistry and physics. 



Repetition 

In both mathematics and science, topics remained in our composite 
U.S. curricula for more grades than all but a few other TIMSS countries. The 
U.S. approach can be characterized as “come early and stay late.” 

In mathematics, the U.S. pracUce is to add far more topics than other 
countries do in grades one and two and then repeat these topics undl grade 
seven. In grades nine and 11 the U.S. composite curriculum drops many 
more topics than other countries. On average, mathematical topics remain in 
the U. S. composite curriculum for two years longer than the international 
median. Only five countries have higher average duradons. 

In science, U.S. pracdce is to introduce new science topics at intervals, 
especially grades one and five, with little change in the intervening grades. 
In grades 10 to 12, the U.S. composite curriculum drops many more topics 
than other countries. Average intended duration is close to the international 
median in earth sciences and life sciences, but the U.S. average duration in 
the physical sciences is two years longer than the median and higher than all 
but seven countries. 
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In mathematics, the tendency to retain topics over many grades may 
reflect the traditional approach of distributed mastery — the idea that master- 
ing pieces of a subject would lead to mastery of a bigger whole. U.S. curricula 
lack a strategic concept of focusing on a few key goals, linking content to- 
gether, and setdng higher demands on students. 

This propensity for inclusion extends even to reform proposals. Many 
reform recommendations simply add to the existing topics (or are imple- 
mented by adding to existing content), thereby exacerbating the existing lack 
of curricular focus. 



Emphasis 

U.S. curricula in mathematics and science seek to do something of 
everything and less of any one thing. Given roughly comparable amounts of 
instructional time, this topic diversity limits the average amount of time 
allocated to any one topic. 

In mathematics, this accumuladon may be a product of our model of 
distributed mastery over the grades. 

The reasons for the better results in science are less clear but seem 
related to general science approaches that move from topic to topic. 



Variations Among States 

While the core of mathematics topics was broad, it varied little among 
the states. 

The number of core science topics was much smaller, and the overlap 
among state curricula was also small. While students in U.S. states might have 
studied a number of science topics roughly equal to the international median, 
the differing curricular intentions of various states are such that students in 
different states likely studied only a few common topics. 



Defining the "Basics" 

Student achievement in mathemadcs and science in any country is largely 
shaped by what educadonal policymakers in that nadon regard as “basic” in 
these subjects and how well they communicate and support those basics. 

The U.S. mathemadcs instructional practices defined de facto eighth-grade 
basics of arithmetic, fractions, and a relatively small amount of algebra. In Ger- 
many, Japan, and internationally, the basics were defined as algebra and geometry. 
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For science, the picture is more complex since U.S. curricula include 
single-area courses, such as physical sciences, life sciences, or earth sciences. 
These courses defined a more restricted, focused set of basics, but they 
applied only to the subset of students receiving those pardcular courses. 



Unfocused Textbooks 

Textbooks play an important role in making the leap from intentions 
and plans to classroom acdvides. They make content available, organize it, 
and set out learning tasks in a form designed to be appealing to students. 
Without restricdng what teachers may choose to do, textbooks drasdcally 
affect what U.S. teachers are likely to do under the pressure of daily instruc- 
tion. The question thus arises: Do U.S. mathematics and science textbooks 
add guidance and focus to help teachers cope with unfocused curricula? 

Unfortunately, the answer is “no.” The splintered character of mathemat- 
ics and science curricula in the United States is mirrored in the textbooks used 
by teachers and students. Textbooks are unfocused in several ways: 



Topics Included 

The U.S. mathematics and science textbooks include far more topics 
than was typical internationally at all three grade levels analyzed. 

In mathematics, U.S. textbooks are far above the 75th percentile of the 
TIMSS countries in the number of topics covered. For example, U.S. math- 
ematics textbooks designed for fourth and eighth graders cover an average of 
30 to 35 topics, while those in Germany and Japan average 20 and 10 respec- 
tively for these populations. 

As a result, typical mathematics textbooks in the United States look quite 
different than those of a nation such as Japan. The typical eighth-grade U.S. 
textbook (non-algebra) is larger and more comprehensive than the average 
Japanese text, but it contains fewer sequences of extended attention to a 
particularly important topic. The U.S. textbooks’ sequences are also shorter 
and have more breaks. The lesson-by-lesson organization of the U.S. book is 
much less focused than the Japanese book, and there is far more skipping 
among topics in successive segments. 

In science, the differences are even greater. At all three population 
levels, U.S. science textbooks included far more topics than even the 75th 
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percentile internationally. The average U.S. science textbook at the fourth, 
eighth, and 12th grade has between 50 and 65 topics; by contrast Japan has 
five to 15 and Germany has just seven topics in its eighth-grade science 
textbooks. 



Emphasis 

The propensity of U.S. curricula to do something of everything but little 
of any one thing is mirrored in textbooks. The few most emphasized topics 
account for less content than is the case internationally. 

Among the fourth-grade mathematics textbooks invesdgated, the five 
topics receiving the most textbook space accounted on average for about 60 
percent of space in the U.S. textbooks but over 85 percent of textbook space 
internationally. 

At the eighth-grade level, the five most emphasized topics in non- 
algebra U.S. textbooks accounted for less than 50 percent of textbook space 
compared to an international average of about 75 percent. An exception is 
U.S. eighth-grade algebra books, which were highly focused, with the five 
most emphasized topics accounting for 100 percent of the books. 

Among the U.S. fourth-grade science textbooks invesdgated, the five 
topics receiving the most attention accounted for an average of just over 25 
percent of total space in U.S. textbooks compared to an average of 70 to 75 
percent internationally. 

Among the U.S. eighth-grade science textbooks invesdgated, the five 
most emphasized topics in more general science texts accounted for about 50 
percent of textbook space compared to an international average of about 60 
percent. In contrast, U.S. eighth-grade science books oriented to a single 
area were highly focused, with the five most emphasized topics accounting 
for more of the textbooks than was true in the international average. 



Difficulty 

U.S. eighth-grade science textbooks emphasized understanding and 
using routine procedures, which represent the less complex, more easily 
taught expectations for student performance. This emphasis was typical of 
what was done internationally. It is not, however, typical of the diverse and 
more demanding performances called for in current U.S. science education 
reform documents. 
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Most U.S. schools and teachers make selective use of textbook contents 
and rarely, if ever, cover all of a textbook's content. Publishers can reasonably 
expect that those who adopt and buy a particular textbook will feel free to use 
only the contents that suit their purposes. Unfortunately, the result is large 
textbooks covering many topics but comparatively shallowly. Even in the 
largest textbooks, space is still limited. It is impossible for textbooks so 
inclusive to help compensate for unfocused official curricula. Thus, our 
analysis shows that U.S. textbooks support and extend the lack of focus seen 
in those official curricula. 




How Teachers Deal with the Splintered Vision 

Teachers in the United States are sent into their classrooms with a 
mandate to implement inclusive, fragmented curricula and armed with 
textbooks that embody the same “breadth rather than depth” approach. How 
do they handle such a situation? Not surprisingly, the instrucdonal decisions 
made by U.S. teachers mirror the inclusive approach of the tools they are 
given. U.S. teachers handle the splintered vision they get in several ways: 

Topics Covered 

U.S. mathematics and science teachers typically report teaching more 
topics than their counterparts in other countries, including Germany and 
Japan. This is true for science teachers even when using a single-area text- 
book such as physical science, life science, or earth science. 

Emphasis 

Since instrucdonal time for science or math within a school year is 
limited, the data show that teaching more topics means that teachers spend 
little Ume on most topics. 

U.S. eighth-grade mathematics teachers indicated that they taught at 
least a few class periods on all but one topic area included in the teacher 
survey's quesdonnaire. They devoted 20 or more periods of in-depth instruc- 
tion to only one topic area, fractions and decimals. However, in Germany and 
Japan many other topic areas received this more extensive coverage. 

According to the survey, the five topic areas covered most extensively by 
U.S. eighth grade mathematics teachers accounted for less than half of their 
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year's instructional periods. In contrast, the five most extensively covered 
Japanese eighth-grade topic areas accounted for almost 75 percent of the 
year's instructional periods. 

U.S. eighth-grade science teachers also indicated that they would devote 
at least some class time to every topic area surveyed. None was omitted 
completely and no topic was marked to receive more than 13 class periods of 
attention by eighth-grade physical and general science teachers. Additional 
topic areas received more extensive coverage in Germany and Japan. 

On average, U.S. eighth-grade general science teachers' most exten- 
sively covered topics accounted for only about 40 percent of their instruc- 
tional periods, but this percentage was also lower for science in Germany and 
Japan (about 50 to 60 percent). 

Number of Activities 

U.S. teachers engage in more teaching activities per lesson than their 
counterparts in other countries. More than 60 percent of U.S. eighth-grade 
mathematics and science teachers reported using six or more activities in a 
typical class. In Germany only 25 percent reported using six or more activi- 
ties, and even fewer reported doing so in Japan. 



Is This The Best Our Teachers Can Do? 

U.S. mathematics and science teachers work hard and often face de- 
manding workplaces. Our data show that they are scheduled to work about 
30 periods each week, which is more than teachers in Germany (just over 20 
periods) and Japan (fewer than 20). 

These teachers rarely have the luxury of being idealists. Unfocused 
curricula and inclusive textbooks set few boundaries for instructional deci- 
sions and appear to require a little bit of everything. It is easier for real 
teachers making real decisions in the real workplaces of U.S. schools to settle 
for the first alternative that seems good enough rather than search for the 
best possible instruction. They try to cover as much as they can rather than 
teach just a little. In a word, they “ satisfice .” 
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The data show that U.S. mathematics and science teachers are aware of 
and believe in more effecdve, complex teaching styles than they practice. 
They often have information that would help them do their work more 
effectively. Their beliefs suggest that they might choose to organize instruc- 
tion differently under circumstances less consumed by the need for coverage. 

Effective teachers should not be unusual, nor should effectiveness 
require extraordinary efforts and dedication by teachers. The reality, how- 
ever, is that U.S. teachers are placed in situations in which they cannot do 
their best. We have yet to unleash the effectiveness of U.S. teachers. It seems 
likely that fundamental changes are needed in teacher knowledge, working 
conditions, curricula quality, student expectations, and textbook content. 



What Can We Expect from U.S. Students? 

In mathematics, we have a highly fragmented curriculum, textbooks that 
are a “mile wide and an inch deep,” and teachers who cover many topics but 
none extensively. We make low demands on students and have a more lim- 
ited concepdon of “the basics” than the international norm. It seems highly 
likely that U.S. student achievement in mathematics will be below interna- 
tional averages. 

Our science curriculum is less fragmented. Science achievement seems 
likely to be closer to international averages, but still not what we desire and 
certainly below some, if not most, of our economic peers. 

U.S. students’ achievements — the yield of our aggregate nadonal educa- 
don “system” — in mathemadcs and the sciences are likely to be disappoindng 
and many of the reasons are not under students’ control. We must make 
substantial changes if we are to compete and to produce a quantitatively and 
scientifically literate workforce and citizenry. 



How Has Our Vision Become So Splintered? 

Culture affects education, even in supposedly fixed disciplines such as 
mathemadcs and science. Countries differ in the priorities they give to these 
disciplines, in the way they organize instruction and the value they ascribe to 
academic success. The qualitative differences found in mathematics and 
science instrucdon across France, Japan, Spain, Switzerland, Norway, and the 
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United States suggest that strong cultural components, even national ideolo- 
gies, are at work in the teaching of these subjects. 

The current state of our nation’s composite visions guiding mathematics 
and science education are clearly shaped by cultural forces particular to the 
United States, starting with the nation’s decentralized approach to education. 

A System With Many Actors 

Education in the United States always has been guided by agencies and 
organizations — local, state, and national, official and unofficial — that take 
their share of responsibility for education. There are many actors, including 
states, schools, commercial publishers, national associations, test publishers, 
teachers, and the federal government. While the independence of these 
groups is essential to education in the United States, the result is a composite 
of sometimes corresponding, sometimes conflicting separate visions. The 
conversations that cumulatively shape the national visions of mathematics and 
science education in the United States appear to be held in the tower of Babel. 

Our earlier statement that there was “no one helm” for mathematics and 
science education should not be taken as implying that there should be either 
a single helm or a single helmsman. At its best, our system of distributed 
educational responsibility allows local preferences and community needs to 
help determine what occurs in local schools. It also provides laboratories to 
test, implement, and replicate new approaches. At its worst, our system 
requires that we seek consensus on needed changes site by site. 

Given the cultural context in which mathematics and science education 
is carried out in the United States, a decentralized system with many actors is 
inevitable. We hope the splintering is not. 

A Diverse Market for Curricula and Textbooks 

U.S. textbook publishers face varied, often conflicting, demands for 
what should be in mathematics and science textbooks. Official mathematics 
and science curricula vary widely among states and districts. Over 35 states 
have textbook adoption policies, but in many states districts are free to 
choose any textbook. 

Textbook publishers are understandably eager to produce products that 
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will appeal to as many markets as possible. The results are large textbooks 
that embrace virtually all suggestions offered by the various actors. They 
include something for everyone. 

If a clear, coherent vision of what is important existed and was shared by 
virtually all textbook publishers, it is likely that the resulting materials could 
soon lead to wide official adopdon reflecdng that coherent vision. 

Standardized Tests 

The cacophony of conflicdng demands seen in curricula and textbooks 
is exacerbated by pressures to provide for successful student performance on 
common standardized tests. These include state assessments and the Na- 
tional Assessment of Educational Progress (NAEP) tests, as well as commer- 
cially produced and locally mandated standardized tests. 

Despite a seeming sameness about most standardized tests, there are differ- 
ences in content emphases and student performance demands. These differences 
are sufficient to constitute yet another set of demands to try to reconcile. 

Mass Production and Mass Education 

U.S. education has been influenced profoundly by a deeply seated ideol- 
ogy springing from our national experience with the power of industrial and 
assembly-line production. This ideology revolves around the idea of producing 
uniform, interchangeable parts that can be assembled into desired wholes. 
Translated into education, such thinking views school mathematics and science 
as partitioned into many topics that form the building blocks of curricula. As a 
result, our students may grasp the pieces but not the whole. 

We have applied the term incremental assembly to this ideology and 
believe that it may well keep the United States from finding other, more 
coherent and powerful ways to think about and organize curricula. This is 
unfortunate. Henry Ford, presumably, did not try to make all models simulta- 
neously on the same assembly line. 

The lack of focus in U.S. mathematics and science education also has 
roots in behavioral psychology, which has pushed education in the direction 
of behavioral objectives and programmed instruction. This notion may help 
explain our curricula of many small topics, frequent low demands, and 
interchangeable pieces of learning to be assembled later. 
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The Impact of Reform 

In the U.S. today we live in a climate of reform and talk of reform. 
Professional organizations concerned with mathematics and science educa- 
tion issue platform documents setting out agendas, benchmarks, and “stan- 
dards.” These powerful, demanding, and insightful calls for reform offer 
coherent visions of what might be done to make major improvements in their 
targeted educational practices. What has been their impact on mathematics 
and science education? 

Awareness of Reforms 

As late as two years ago, state mathematics and science curriculum 
guides, plans, and statements of intendons still called for coverage of far 
more topics than most other countries did and, far more than would be 
indicated by current reform agendas in mathematics and science educadon. 
The same can be said of textbooks. 

U.S. mathematics and science teachers are generally aware of these 
reforms. More than 75 percent of U.S. mathemadcs teachers indicated 
familiarity with the NCTM Standards. Fewer U.S. science teachers indicated 
similar familiarity with the corresponding science frameworks, but those were 
released five years after the mathematics report 

These data suggest that more time alone will not be enough. Failure to 
create widespread change in teaching practice does not appear to be due to 
lack of informadon. 

An Unfocused Reform 

U.S. mathemadcs and science educators approach reform in the same 
inclusive style as they deal with tradidonal content — they add its recommen- 
dations but do not take away. This is clearly contrary to the recommenda- 
tions of the NCTM Standards. Textbooks have been affected to some extent 
by mathematics education reform recommendations, but in a similar inclusive 
manner. As a result, students cannot focus on or be successful in either the 
old or new curricula. 

This development is troubling because the reform agendas typically are 
coupled with more demanding, time-consuming, and complex performance 
expectadons that require paying careful attendon to a smaller number of 
strategic topics. Adding more topics will not help. 
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The impact of science reform recommendations remains untested as yet. 
Reform documents themselves often emerge from compromise among 
professionals, and this compromise may prevent them from stadng a single 
vision. Even when a reasonably coherent voice emerges for reform — for 
example, the NCTM Standards in math educadon and the National Research 
Council’s National Science Education Standards — our organizational context 
causes it to be heard as simply one more voice in a “babel” of compedng 
voices. This “babel” becomes so overwhelming that it is hard for official 
actors to separate “signal” from “noise” or to prioridze the voices to which 
they will listen. In such a systemic context, splintered visions are likely to 
remain splintered. 

The Need for Time 

The lack of success that reform measures have had to date does not 
imply that they have been futile. Rather, they imply that reform may take 
considerable time. 

Slow progress is certainly no reflection on the quality or power of math- 
ematics and science reform efforts that have yet to be as effectual as their 
supporters wish in this climate. Certainly, it would be drastically wrong to 
conclude, based on these “early returns,” that these reforms have failed. 
Rather, it seems more appropriate to be amazed at their current successes. 



Who Are Our Curricular Peers? 

If we take seriously that the propordons of curricula, as set forth in state 
guidelines and textbooks, set bounds on what is broadly achieved by those 
taught, we should identify those countries that set similar bounds to their 
students’ achievement. 

In grade-eight mathematics, the U.S. composite curriculum as repre- 
sented by textbooks is most like those of Australia, New Zealand, Canada, 
Italy, Belgium (French language system), Thailand, Norway, Hong Kong, 
Ireland, and Iceland. 

In grade-eight science, our curriculum is most like that of New Zealand, 
Iceland, Greece, Bulgaria and the Peoples’ Republic of China. 
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Are These the Peers We Want? 

While the curriculum of any country is interesting and has some impor- 
tant features, we must ask if these are the countries with whom we are and 
will be trying to compete. 

As a nation we desire to empower and inform our citizenry comparably 
as well as to effectively compete economically with other developed countries. 
We want attainments similar to the European Union, to the APAC countries 
(especially Japan and the “young tigers” of Korea, Singapore, etc.), and, most 
definitely, with the other G-7 countries. 

When we find ourselves most similar to countries other than those with 
whom we seek to be peers, we have reason for deep concern. In matters of 
what is basic in teaching children mathematics and science, we are not peers 
with the composite of other TIMSS countries. We as a nation must be con- 
cerned. 



What Is Necessary for Reform to Succeed? 

The U.S. vision of mathematics and science is splintered. We are not 
where we want to be. We must change. But the required change is fundamen- 
tal and deeply structural. There are no single answers or instant solutions. 

Most nations do not share similarly splintered visions in mathematics 
and science education. Theirs are more coherent. While central guiding 
visions do not alone guarantee student achievement, they contribute to 
optimal attainments. These shared visions are insufficient to ensure desired 
achievements, but they seem necessary starting points. 

The United States has a decentralized educational system in which the 
component organizations do not always work towards common goals, nor do 
they always aim at producing important combined results. Formal mecha- 
nisms of coordination — either by regulation or rewards for selected behav- 
iors — have proven politically sensitive and are in limited use. Given such a 
culture of education, how can a focused vision be achieved? Several principles 
would seem to be at work: 
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Effective Reform Will Necessarily Be Systemic 

Information- and motivation-based reforms and improvement policies 
alone will not bring fundamental improvements. Any serious attempt at 
change in U.S. science and mathematics education must be deeply structural. 
The fundamental problem is not a conglomeration of individual problems. 
Any effective reform in this context will necessarily be systemic — affecting 
several parts at once. 

Not every systemic reform automatically will address the core of our 
problems. Appropriate structural reform must pursue focused and meaning- 
ful goals. We may not be able to do everything and do any of it well. Instead, 
it appears we must make choices regarding which goals are more important 
and how many goals we can effectively pursue. 

Effective Reform Must Respect Cultural Context 

Whatever acdons are taken must be appropriate to our educational 
federalism and our context of shared educational responsibility. When 
discussion suggests the need for more powerful and coherent guiding visions, 
they must be sought in processes that will achieve wide consensus necessary 
for change in our context. 

A corollary is that we may learn from other countries but we cannot 
emulate their centrally administered changes. Any reform in the United 
States must seek visions that can achieve broad consensus. 



Conclusion 

The United States needs powerful mathematics and science education 
because they: 

■ Provide a strong basis for our democracy by helping create a literate 
and informed citizenry; 

■ Help each individual to grow, develop, reach his or her individual 
potential, and become more autonomous and empowered; and 

■ Provide a sound basis for continuing national prosperity in a com- 
petitive, informadon-driven, technological, and changing interna- 
tional arena. 
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Perhaps we do not need a central focus for our curricula and teaching. 
Perhaps the value of diversity outweighs the value of focus. Perhaps our de 
facto emphasis on breadth will prove more effective overall than other coun- 
tries’ strategies of focusing on strategic topics. That is a matter for further 
empirical evidence and public discussion. 

Both conventional wisdom and a considerable body of research, how- 
ever, suggest that focus and selection are needed in situations in which too 
much is included to be covered well. The impact of these unfocused curricula 
and textbooks in mathematics and science likely includes lower “yields” from 
mathematics and science education in the United States Focus would seem to 
be a necessary but not a sufficient condition for high student attainments. 

What kinds of mathematics and science education do we, as a nation, 
want for our children? While this is a central quesdon for public debate, it 
seems likely that we want educations that: 

■ Are more focused, especially on powerful, central ideas and capacides; 

■ Provide more depth in at least some areas, so that the content has a 
better chance to be meaningful, organized, linked firmly to 
children’s other ideas, and to produce insight and intuition rather 
than rote performance; and 

■ Provide rigorous, powerful, and meaningful content that is likely to 
produce learning that lasts and not just learning that suffices for the 
demands of schooling. 



Questions to Ask 

The authors of this report do not represent any official or policymaking 
group. Our job has been to design relevant research, analyze its results 
carefully, and report them objectively. Because of who we are, we do not feel 
it appropriate to make specific recommendations. We can, however, at least 
ask quesdons — quesdons that our results lead us to believe important for 
those who do set policy. 

Most of these quesdons are not original with us, although their form 
here has been influenced by the data we investigated. In fact, some efforts are 
currently underway to address these questions, including the National Sci- 
ence Foundadon’s State Systemic Initiadves and the recently convened 
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executive committee of the National Governors’ Association in conjunction 
with business leaders. Those efforts may include answers to several pressing 
questions raised by these findings: 

■ How can we focus our mathematics and science curricula and text- 
books around an intellectually coherent vision? 

■ How can we raise expectations and demands on our students? 

■ How can we help our teachers to do the best they can in teaching 
mathematics and science to our students? 

■ How can we find a better model for curriculum and instruction? 

■ How can we develop a new vision of what is basic and important? 

Certainly these are not the only questions that must be asked and an- 
swered on the way to the revolution or, if one prefers, to a fruitful evolution 
in mathematics and science education. We have not touched on whole do- 
mains of issues — for example, concerns for equity in educational opportu- 
nity — because we did not want a report on the “splintered vision” of our 
children’s education to be itself unfocused. Others must join in seeking 
answers to the questions raised here and the others we did not raise. Our 
data can help. 

Presently, however, our story is simple. The U.S. vision of mathematics 
and science education is splintered. We are not where we want to be. We must 
change. 
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leased. Therefore this report cannot offer any conclusions about these re- 
forms. 
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Using a hypothetical teacher, Ms. Lane, the author notes the various 
difficulties she might face in teaching a unit on light and vision. The 
four problem areas identified are (1) definitions of strategic teaching, 

(2) course curriculum, (3) student learning, and (4) student instruction. 
The author elaborates on the difficulties faced in each area and exam- 
ines possible teaching responses. A teaching strategy based on concep- 
tual change learning was found to be the most effective. Recommenda- 
tions for improving teachers’ ability to utilize strategic thinking and 
conceptual change learning are offered. 
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Washington, DC: U.S. Department of Education, 1996. 

A study of curriculum reform in mathematics, science, and higher order 
thinking. The study examined emerging reform practices focusing on 
which ones worked; what incentives for change were effective; and which 
means of overcoming barriers to change were successful. Nine case 
studies were conducted at various sites where reform was in progress. 
Preparatory work, reform methods, and results are discussed. 
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Teachers’ Perceptions of Autonomy and Satisfaction.” Educational Evaluation 
and Policy Analysis , 16(1), Spring 1994, pp. 21-39. 

A study designed to determine the effects of centralized curriculum 
control on teachers’ opinions. High school mathematics and social 
studies teachers from six urban school districts were asked to rate their 
control and autonomy in various contexts. The districts were divided by 
type into high, medium, and low levels of centralized curriculum con- 
trol. Mean ratings were presented from teachers in all three levels of 
centralization. The four scales were (1) Policy and Teacher-Discretion 
Influences, (2) Teacher Control Over Classroom Content and Pedagogy, 
(3) Staff Involvement in Decision Making for Course Content, and (4) 
Teacher Empowerment. Results were discussed in terms of individual 
scale ratings, overall ratings, and comparisons between mathematics and 
social studies teachers. 



Ball, D.L., and Cohen, D.K. “Reform by the Book: What Is — Or Might Be — 
The Role of Curriculum Materials in Teacher Learning and Instructional 
Reform?” Educational Researcher , Volume 25, December 1996, pp. 6-8. 
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A discussion of the impact of curriculum materials on classroom instruction. 
The article suggests that the design of curriculum materials is one of the 
oldest ways to influence classroom instruction. Reformers have often used 
instructional materials as a means to shape what students learn. This strategy 
is often unsuccessful for it undermines the professional work of teaching and 
severely limits local discretion over curriculum. Curriculum materials play an 
“uneven role” in teaching practice because: (1) curriculum developers and 
others often have failed to take account of the teacher; (2) a gap exists be- 
tween curriculum developers’ intentions for students and teachers’ under- 
standing of the material and their beliefs about what is important for students 
to learn; and (3) many educators do not follow textbooks, but instead make 
their own curriculum. Hostility to texts has inhibited careful consideration of 
the constructive role that curriculum might play. The authors suggest that 
curriculum materials could lead to improved practice if they were created 
with closer attention to processes of curriculum enactment. 
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Brophy, J., and Alleman, J. “Activities as Instructional Tools: A Framework 
for Analysis and Evaluation.” Educational Researcher , Volume 20, May 1991, 
pp. 9-23. 

A study designed to stimulate increased scholarly attention to the de- 
sign, selection, and assessment of learning activities and scaffolding of 
student involvement in those activities. It is argued that research has 
focused much more on content selecdon and representation and on 
teacher-student discourse than on activities. A conceptual analysis and a 
set of principles are proposed as tools for designing, selecting, and 
evaluating learning acdvities. The principles were drawn from generic 
curriculum sources and thus, theoretically, are applicable to all school 
subjects. Five sets of principles are offered: (1) primary principles that 
must apply to individual activities; (2) secondary principles that may 
apply to individual activities; (3) principles that apply to sets of activides; 
(4) additional principles that are associated with particular philosophies; 
and (5) principles that apply to the teacher’s implementadon of activi- 
ties. The suggested principles are intended to help students engage in 
actively processing curriculum content, developing personal ownership 
and appreciation of it, and applying it to their lives in and outside of 
school. 

Eggebrecht, John; Dagenais, Raymond; Dosch, Don; Merczak, Norman J.; 
Park, Margaret N.; Styer, Susan C.; and Workman, David. “Reconnecdng the 
Sciences.” Educational Leadership , 53(8), May 1996. 

A description and discussion of the Integrated Sciences program at the 
Illinois Mathemadcs and Science Academy. Designed to bring all sci- 
ence subjects together into one classroom, the program is a three- 
semester, double-period course offered as an alternadve to standard 
core sequence science instrucdon. The program is constandy evolving 
and adopdng new techniques of instruction. By integrating the varied 
subjects, the instructors hope to overcome the deficiencies of standard 
science educadon: transfer of basic sciendfic knowledge and transfer of 
sciendfic authority. According to the authors, the program must be 
assessed in five key areas: (1) the context of national standards; (2) 
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student performance in relation to those standards; (3) positive student 
attitudes toward science; (4) adequate preparadon for advancement; and 
(5) overall improved learning habits. Results indicated a positive out- 
come in the first four areas, while the fifth was sdll being determined. 
The program has demonstrated its success and has led to real intellec- 
tual growth for both students and instructors. 

Goertz, Margaret E.; Floden, Robert E.; and O’Day, Jennifer. Studies of 
Education Reform: Systemic Reform. Washington, DC: U.S. Department of 
Education, 1996. 

A study of systemic reform in 12 schools. The study was designed to 
expand knowledge of state approaches to systemic reform, examine 
responses to reform policies, identify challenges at all levels to reform- 
ing education, study the capacity of the system to support reform, and 
provide guidance to policymakers at all levels as they design and imple- 
ment reform policies. A discussion of relevant literature is included. 
Analysis was done in three areas: (1) challenges to implementing sys- 
temic reform; (2) teachers’ practice and opportunities to learn; and (3) 
capacity building and systemic reform. Policy and research implications 
are also offered. 

Khattri, Nidhi; Reeve, Alison L.; Kane, Michael B.; and Adamson, Rebecca J. 
Studies of Education Reform: Assessment of Student Performance. Washington, DC: 
Pelavin Research Institute, 1996. 

A study of performance assessments from various educadonal authorides 
at 16 school sites. The main objecdves of the study were to examine the 
key characterisdcs of performance assessments, the facilitators and barri- 
ers in assessment reform, and the impacts of performance assessments on 
teaching and learning. Results were discussed as general policy implica- 
dons, policy implicadons if assessment reform is to be used to improve 
and inform instrucdon and curriculum, and policy implicadons if assess- 
ment reform is to be used for school and/ or district accountability. A 
discussion of implicadons for future research is also included. 
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Porter, Andrew. “A Curriculum Out of Balance: The Case of Elementary 

School Mathematics.” Educational Researcher, 18(5), June July 1989, pp. 9-15. 

A study to examine the content of elementary mathemadcs courses. 
Using teacher logs and interviews, the researchers determined that 
teachers overemphasized skill development while neglecting to develop 
adequate conceptual understanding and problem-solving abilities. They 
also found that the majority of topics were taught briefly with no expec- 
tadon of in-depth understanding. The differences in content between 
grades were negligible and did not align with student progress. Indi- 
vidual teachers and schools varied enormously in the overall amount of 
time spent teaching mathematics. Recommendations for correction are 
offered. 

Porter, Andrew. “The Uses and Misuses of Opportunity-to-Learn Standards.” 

Educational Researcher, 24(1), January-February 1995, pp. 21-27. 

A discussion of Opportunity-to-Learn (OTL) standards as defined by the 
Goals 2000: Educate America Act. Noting the complicated legislative 
and political atmosphere surrounding the law, the author speculates on 
possible uses and definitions for OTL standards. The author states that 
OTL standards are not appropriate as measures of school accountability, 
but rather as measures of school improvement. It is noted that the 
standards can be used to determine successful practices, process indica- 
tors, and reform progress and as a tool to diagnose poor performance. 
OTL standards should include first and foremost a basic standard of a 
safe and orderly environment. What is actually taught in schools; the 
quality of pedagogy used; and the applications and uses of instructional 
resources such as libraries, laboratories, and computers should also be 
included. Criteria suggestions are also offered for other standards. 
While Goals 2000 may not fulfill all expectations, OTL indicators can be 
extremely helpful in aiding school improvement. 
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Roth, Kathleen J. “Science Education: It’s Not Enough to ‘Do’ or ‘Relate.’” In 
Marcia K. Pearshall, ed .In Scope , Sequence , and Coordination of Secondary School 
Science, Volume II: Relevant Research. Washington, DC: The National Science 
Teachers Association, 1992, pp. 151-164. 

A discussion and evaluation of elementary science education teaching 
perspectives. In addition to traditional textbook-based science teaching, 
the author compares and contrasts the three main reform movements in 
science education. Inquiry teaching emphasized process skills and 
actual scientific work. Science-Technology-Society teaching also pro- 
moted process skills, but regards the use of scientific knowledge as most 
important. Conceptual change teaching involved imparting conceptual 
understanding of science and the overall usefulness of scientific knowl- 
edge. The final perspective was found to be the strongest. 



Schmidt, William H.; McKnight, Curtis C.; Raizen, Senta A. A Splintered 
Vision: An Investigation of U.S. Science and Mathematics Education. Boston, MA: 
Kluwer Academic Publishers, 1996. 

A Splintered Vision discusses data from the analysis of 491 curriculum 
guides and 628 textbooks from around the world collected as part of 
TIMSS. It also presents detailed accompanying data on teacher prac- 
tices in the United States, Japan, and Germany. The report documents 
and characterizes the state of U.S. mathematics and science curricula, 
textbooks, and teaching practices and places them in a cross-national 
context. The report’s authors conclude that the lack of a single coher- 
ent vision of how to educate today’s children produces unfocused cur- 
ricula and textbooks that influence teachers to implement diffuse learn- 
ing goals. 
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Smith, Edward L.; Blakeslee, Theron D.; and Anderson, Charles, W. “Teach- 
ing Strategies Associated with Conceptual Change Learning in Science.” 
Journal of Research in Science Teaching ; 30(2), 1993, pp. 111-126. 

A study examining conceptual change learning strategies for science 
teaching and its relationship to student performance. Thirteen seventh- 
grade life science teachers were observed teaching three units: photo- 
synthesis, cellular respiration, and matter cycling in ecosystems. The 
teachers were divided into three groups; each group received varying 
levels of conceptual change learning preparation in the form of work- 
shops and/or course materials. Students were given both pretests and 
posttests. The results demonstrated that the use of these instructional 
materials increased conceptual change learning and resulted in better 
student performance on posttests. Workshops alone had little impact on 
either area. A combination of both strategies was associated with higher 
student performances on tests. The study demonstrates the effective- 
ness of conceptual change learning strategies; however, few of the 
teachers were successful in implementing the strategies without the 
course materials. A discussion of conceptual change theories is also 
included. 

U.S. Department of Education. National Center for Education Statistics. 
Pursuing Excellence: A Study of U.S. Eighth-Grade Mathematics and Science Teach- 
ing , Learning, Curriculum , and Achievement in International Context , NCES 97- 
198, by Lois Peak. Washington, DC: U.S. Government Printing Office, 1996. 

A report on the U.S. eighth-grade TIMSS results. It draws from the 
assessments, surveys, video, and case studies of TIMSS to summarize the 
most important findings concerning U.S. achievement and schooling in 
international context. 



j 88 

o 

__E KLC 

a»g i asr«, i ij GUIDEBOOK TO EXAMINE SCHOOL CURRICULA 



185 



ATTAINING EXCELLENCE: TIMSS AS A STARTING POINT TO EXAMINE U.S. CURRICULA 



U.S. Department of Education. National Center for Education Statistics. 
Pursuing Excellence: A Study of U.S. Fourth-Grade Mathematics and Science , NCES 
97-255, Washington, DC: U.S. Government Prindng Office, 1997. 

A report on the U.S. fourth-grade TIMSS results. It summarizes the 
most important findings concerning U.S. achievement and schooling in 
international context. 
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